Software Testing Basics

Product Managers, you don't need to be a statistician to run good A/B tests. All you need to do is understand the basics. Here is a simplified explanation of A/B tests. 👉What is it 👉Example 👉Why do we do it 👉Things to be careful about 👉What else 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗶𝘁 Every time you're deciding between two options, but don't know which one will work better, A/B testing comes in. It’s a simple way to test two options and find out which one delivers better results—using real data, not intuition. 𝗘𝘅𝗮𝗺𝗽𝗹𝗲 Let's say you’re sending a critical email, and want more and more people to open it (because if they open it, they will read it, if they read it, they will buy your product.) And to get more readers to open your email, you're considering two options (A and B) for the subject line. A: is direct and to the point. B: is creative and catchy. Now you could guess, which one will do the job. You could ask your friends, peers, which on they like. Or you could test both: You send Option A to one half of your audience and Option B to the other. Then, you wait and see which one led to higher opens. Simple! You've just run an A/B test 𝗪𝗵𝘆 𝗗𝗼 𝗪𝗲 𝗗𝗼 𝗜𝘁? Because the cost of guessing wrong can be huge. When these decisions could impact millions of users and lead to significant changes in revenue, you don't want to be GUESSING!!! It's always better to have data. 𝗧𝗵𝗶𝗻𝗴𝘀 𝘁𝗼 𝗯𝗲 𝗰𝗮𝗿𝗲𝗳𝘂𝗹 𝗮𝗯𝗼𝘂𝘁 𝘄𝗵𝗶𝗹𝗲 𝗔/𝗕 𝘁𝗲𝘀𝘁𝗶𝗻𝗴 ■ 𝗦𝗮𝗺𝗽𝗹𝗲 𝘀𝗶𝘇𝗲 𝗺𝗮𝘁𝘁𝗲𝗿𝘀: if you don't test with the right no. of users, your results could mislead you and make you choose the wrong option. (Calculating the right no. of users can be done mathematically. And there are a lot of third party tools that can do it for you!) ■ 𝗧𝗲𝘀𝘁 𝗼𝗻𝗲 𝘁𝗵𝗶𝗻𝗴 𝗮𝘁 𝗮 𝘁𝗶𝗺𝗲: If you’re testing more than one element at the same time, it will be hard to know what caused the improvement (or the failure). (If Google had tested the color and font size together, they would've not known what led to the success.) ■ 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 𝗺𝗮𝘁𝘁𝗲𝗿𝘀: It is important to run your tests for the right amount of time (which depends on the number of users, the type of tests, and other variables.) Most experimentation tools will do this math for you and tell you how long to run the test ■ 𝗔𝗹𝗹 𝗲𝗹𝘀𝗲 𝗲𝗾𝘂𝗮𝗹: Ensure that every other element remains constant except the one you're testing. For example, if you're testing two landing pages, everything from the time of day to the audience must be the same. ■ 𝗔𝘃𝗼𝗶𝗱 𝗯𝗶𝗮𝘀𝗲𝘀: Be sure to avoid any biases while selecting the audience or running the test. If one version only has a specific group of users, the results could be skewed by their unique behavior patterns. 𝗪𝗵𝗮𝘁 𝗲𝗹𝘀𝗲 A/B testing isn’t just for big companies with millions of users. It could be for any product or decision that affects how people interact with your product/business.

Automation is more than just clicking a button While automation tools can simulate human actions, they don't possess human instincts to react to various situations. Understanding the limitations of automation is crucial to avoid blaming the tool for our own scripting shortcomings. 📌 Encountering Unexpected Errors: Automation tools cannot handle scenarios like intuitively handling error messages or auto-resuming test cases after failure. Testers must investigate execution reports, refer to screenshots or logs, and provide precise instructions to handle unexpected errors effectively. 📌 Test Data Management: Automation testing relies heavily on test data. Ensuring the availability and accuracy of test data is vital for reliable testing. Testers must consider how the automation script interacts with the test data, whether it retrieves data from databases, files, or APIs. Additionally, generating test data dynamically can enhance test coverage and provide realistic scenarios. 📌 Dynamic Elements and Timing: Web applications often contain dynamic elements that change over time, such as advertisements or real-time data. Testers need to use techniques like dynamic locators or wait to handle these dynamic elements effectively. Timing issues, such as synchronization problems between application responses and script execution, can also impact test results and require careful consideration. 📌 Maintenance and Adaptability: Automation scripts need regular maintenance to stay up-to-date with application changes. As the application evolves, UI elements, workflows, or data structures might change, causing scripts to fail. Testers should establish a process for script maintenance and ensure scripts are adaptable to accommodate future changes. 📌 Test Coverage and Risk Assessment: Automation testing should not aim for 100% test coverage in all scenarios. Testers should perform risk assessments and prioritize critical functionalities or high-risk areas for automation. Balancing automation and manual testing is crucial for achieving comprehensive test coverage. 📌 Test Environment Replication: Replicating the test environment ensures that the automation scripts run accurately and produce reliable results. Testers should pay attention to factors such as hardware, software versions, configurations, and network conditions to create a robust and representative test environment. 📌 Continuous Integration and Continuous Testing: Integrating automation testing into a continuous integration and continuous delivery (CI/CD) pipeline can accelerate the software development lifecycle. Automation scripts can be triggered automatically after each code commit, providing faster feedback on the application's stability and quality. Let's go beyond just clicking a button and embrace automation testing as a strategic tool for software quality and efficiency. #automationtesting #automation #testautomation #softwaredevelopment #softwaretesting #softwareengineering #testing

founder learnings! part 8. A/B test math interpretation - I love stuff like this: Two members of our team (Fletcher Ehlers and Marie-Louise Brunet) - ran a test recently that decreased click-through rate (CTR) by over 10% - they added a warning telling users they’d need to log in if they clicked. However - instead of hurting conversions like you’d think, it actually increased them. As in - Fewer users clicked through, but overall, more users ended up finishing the flow. Why? Selection bias & signal vs. noise. By adding friction, we filtered out low-intent users—those who would have clicked but bounced at the next step. The ones who still clicked knew what they were getting into, making them far more likely to convert. Fewer clicks, but higher quality clicks. Here's a visual representation of the A/B test results. You can see how the click-through rate (CTR) dropped after adding friction (fewer clicks), but the total number of conversions increased. This highlights the power of understanding selection bias—removing low-intent users improved the quality of clicks, leading to better overall results.

Wonder why everyone’s talking about ‘shifting testing left’ these days? Shift-left testing means incorporating QA activities earlier in the development process, reducing late-stage defects and rework. By bringing testers into story refinement, architectural discussions, and even design brainstorming, you catch potential issues before they manifest in code. It’s easier to tweak a specification or clarify a requirement than to refactor a tangled implementation. This early involvement makes testers partners in preventing defects rather than mere finders of bugs after the fact. It fosters a culture of shared ownership, where every team member is accountable for quality from the get-go.

Earth Pit Electrode Testing in a Substation. In a substation, the earth pit (grounding electrode) plays a crucial role in ensuring electrical safety by providing a low-resistance path for fault currents. Testing is divided into: 1. Grid Resistance Testing: This test measures the overall resistance of the substation grounding grid. The grounding system is connected in parallel to ensure low resistance, redundancy, reliability, even current distribution, and reduced voltage rise. 2. Individual Earth Pit Testing: In this test, the earth pits are disconnected from the substation grid, and the resistance of each electrode is measured. Steps for Conducting Individual Earth Pit Testing Using the Fall-of-Potential Method: 1. Disconnect the earth pit from the grounding grid to measure its standalone resistance. 2. Place the test electrodes: ▫️Current electrode (C2): 30–50 meters away from the earth pit. ▫️Potential electrode (P2): 10–15 meters away. 3. Connect the earth resistance tester: ▫️Earth electrode (E): Connect to the earth pit under test. ▫️Potential electrode (P2): Measures the voltage drop. ▫️Current electrode (C2): Injects the test current. 4. Run the test and take multiple readings at different distances of P2. 5. Check the resistance value: The general acceptable resistance for an individual earth pit is ≤ 5 ohm. The acceptable resistance for a substation grounding system is ≤ 1 ohm. The image below shows a Uni-T UT521 digital earth tester used to carry out test on 50 earth pits, including 16 earth pits in the transformer yard of a substation.

🚨 Your A/B test results are not the real impact. A happy PM runs an A/B test → sees a +15% lift in revenue → scales the feature to all users → shares the big win in Slack 🎉 But… once the feature is fully rolled out, the KPI impact isn’t there. Why? Because test results often don’t reflect the true long-term effect. Here are a few reasons why this happens: 1️⃣ Confidence intervals matter → That “+15%” is actually a range. The lower bound might be close to zero. 2️⃣ Novelty effect → Users are excited at first, but the effect fades as they get used to the change. 3️⃣ Experiments aren’t additive → Three +15% lifts don’t stack to +45%. There’s a ceiling, and improvements often cannibalize each other. 4️⃣ Sample ≠ population → The test group might not represent your entire user base. For example, you have more high-intent users in the variant. 5️⃣ Time-to-KPI effects → We see that a lot, especially in conversion experiments. The experiment could improve the time to conversion, so when you close the experiment, it seems like you’re winning, but actually if you monitor the users a few days/weeks after the experiment ends, there are no differences in total conversions between the variant and the control. 6️⃣ Type I error → With P-value=0.05 (or worse, 0.1), there’s still a decent chance the “win” is a false positive. 👉 That’s why tracking post-launch impact is just as important as running the experiment itself. Methods like holdout groups, simple correlation tracking, or causal inference models (building synthetic control) help reveal the real sustained effect.

The Three Core Methodologies for Directors of QA 1. Testing is an Opportunity, Not a Metric Too many teams approach testing as a percentage to achieve rather than a system to understand. A function is called inside a test, coverage goes up, and leadership assumes the system is stable. In reality, if testing is not designed to interrogate the software—pushing its limits, exploring its failure points, and ensuring rational behavior—then it is nothing more than an illusion of safety. The best testers do not simply react to defects; they design testing strategies that shape the system itself. They recognize that all testing—unit, integration, and end-to-end—is just exercising the same code. The farther you get from the source, the harder testing becomes. The best QA leaders push for deeper, more effective testing at the earliest layers so that quality is built into the system rather than patched on later. 2. Quality is a Customer-Facing Role Quality assurance is not just about finding defects—it is about delivering confidence. That means QA leaders must be deeply connected to customers. Too often, QA is isolated from the real-world impact of their work, focusing on test cases rather than outcomes. The best QA teams actively engage with users, product teams, and customer success teams to understand: - What actually matters to customers? - Where do users experience pain? - Which failures would erode trust in our product? QA leaders should be part of customer calls, reviewing support tickets, and understanding how real users interact with the software. Without this direct connection, testing becomes disconnected from reality, focusing on low-priority issues while critical customer pain points remain unaddressed. 3. Quality Gets a Seat at the Table For too long, QA has been treated as a secondary function—brought in at the end of the process to “approve” a release rather than being involved in its creation. This approach is fundamentally flawed. The best QA leaders insist on being part of the conversation from day one. They are involved in design discussions, influencing architecture, and shaping processes to prevent defects rather than just detecting them. When QA has a seat at the table: - Testing strategies are built into development rather than added as an afterthought. - Quality is proactive, not reactive. - Teams shift from firefighting production issues to preventing them in the first place. Quality is not a function that simply exists to serve engineering—it is a business-critical discipline that impacts customer trust, retention, and revenue. When QA is involved in strategy discussions, the entire company benefits from a culture of quality rather than just a process of testing. When QA is seen as an opportunity to shape the system, a direct line to customer experience, and an integral part of leadership discussions, it becomes one of the most valuable functions in the company. #QualityEngineering #QA #SoftwareTesting

You're in a Senior Machine Learning Interview at Google DeepMind. The interviewer sets a trap. They hand you a dataset with 15% missing values in the "Age" column and ask a simple question: "How do you handle these missing values before we start training?" 90% of candidates walk right into the trap. The candidate immediately grabs the whiteboard marker. "Easy. I'll calculate the median of the 'Age' column to handle outliers, then fill the empty cells with that value." They might even write the Pandas equivalent:     𝘥𝘧['𝘢𝘨𝘦'] = 𝘥𝘧['𝘢𝘨𝘦'].𝘧𝘪𝘭𝘭𝘯𝘢(𝘥𝘧['𝘢𝘨𝘦'].𝘮𝘦𝘥𝘪𝘢𝘯()) The interviewer nods, smiles, and ends the interview 5 minutes later. They didn't get the job. Why did they fail? Because they treated data cleaning as a generic "pre-processing" step. By calculating the median on the entire dataset (before splitting), they effectively "snooped" on the test set. Their model learned a distribution that includes data it shouldn't see until inference. In the real world, they don't have access to tomorrow's data to calculate today's median. ----- 𝐓𝐡𝐞 𝐒𝐨𝐥𝐮𝐭𝐢𝐨𝐧: To pass this bar, you need to treat your pipeline like a time machine. You must protect the "future" (test set) from the "past" (training set). The engineering reality: - 𝐒𝐩𝐥𝐢𝐭 𝐅𝐢𝐫𝐬𝐭: Divide your data into Train and Test immediately. - 𝐅𝐢𝐭 𝐨𝐧 𝐓𝐫𝐚𝐢𝐧: Calculate your statistics (mean, median, std dev) only using the training set. - 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦 𝐁𝐨𝐭𝐡: Apply those specific training values to fill gaps in both the Train AND the Test set. If your training median is 34 and your test median is 41, you fill the test set holes with 34. 𝐓𝐡𝐞 𝐀𝐧𝐬𝐰𝐞𝐫 𝐓𝐡𝐚𝐭 𝐆𝐞𝐭𝐬 𝐘𝐨𝐮 𝐇𝐢𝐫𝐞𝐝: "I never calculate global statistics. I fit my imputers on the training set and apply that transformation to the test set to prevent data leakage. My pipeline must emulate the blindness of production." #MachineLearning #MLEngineering #DataScience #DeepLearning #MLInterview #DataLeakage #MLOps

I've been in test automation for 20 years. If I were starting my SDET career today — in 2026, with AI changing everything — here are the 5 skills I'd learn first: 1. Playwright over Selenium. Selenium still works. But Playwright is faster, more reliable, and what most modern teams are adopting. Learn the tool companies are hiring for tomorrow, not yesterday. 2. API testing before UI testing. 80% of bugs I've seen in production were API-level issues that UI tests would never catch. Start with Postman, then move to automated API test suites. This is where SDETs make their biggest impact. 3. CI/CD pipeline literacy. If you can't plug your tests into a GitHub Actions or Jenkins pipeline, you're not an SDET — you're a script writer. Understand how your tests run in the real world. 4. AI-assisted test generation. Claude, Copilot, Cursor — learn to use AI to write test scaffolding faster. But more importantly, learn to validate what AI generates. The SDET who can use AI as a tool while catching its mistakes is worth double. 5. Business logic thinking. The hardest skill to teach. Don't just test that the button works. Test that the system behaves correctly when unexpected things happen. This is the skill that separates a £35k QA from a £65k SDET. Notice what's NOT on this list? A computer science degree. Every one of these skills can be learned in 6–9 months with the right structure. Want to know which of these you should prioritise based on your background? I do free 30-minute strategy calls exactly for this. Book yours — link in my profile. #SDET #TestAutomation #Playwright #APITesting #CICD #TechSkills #CareerGrowth #QualityEngineering

𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗞𝗲𝘆 𝗥𝗼𝗹𝗲𝘀 𝗶𝗻 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗮𝗻𝗱 𝗧𝗵𝗲𝗶𝗿 𝗖𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻𝘀 𝘁𝗼 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗔𝘀𝘀𝘂𝗿𝗮𝗻𝗰𝗲 ✴️ 𝗦𝗗𝗘𝗧 - Software Development Engineer in Test: 𝗦𝗗𝗘𝗧s are skilled in both development and testing. They write automated test scripts, develop testing frameworks, and often work on building tools that improve the testing process. They are also involved in writing unit tests, integration tests, and performance tests. - 𝗖𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻: 𝗦𝗗𝗘𝗧s bridge the gap between development and testing by automating repetitive tasks, enabling faster feedback loops, and ensuring the software meets both functional and performance standards. 𝗤𝗘 - Quality Engineer: Quality Engineers focus on the quality processes across the software development lifecycle. They are responsible for defining quality standards, monitoring metrics, and ensuring that quality is maintained throughout. They often look at testing holistically and incorporate both manual and automated strategies. - 𝗖𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻: QEs work to ensure that quality is "baked into" the development process rather than just tested at the end. By setting up proper processes, they help prevent defects early, improve collaboration across teams, and maintain high standards for the overall product quality. 3. 𝗧𝗘 - Test Engineer: Test Engineers are mainly involved in executing tests both manual and automated to find bugs. They focus on ensuring that the product meets the specified requirements. They design and execute test cases, document defects, and verify fixes. - 𝗖𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻: TEs play a critical role in finding defects before release. By executing well-designed test cases, they validate the functionality, usability, and stability of the product, helping to ensure that it meets user expectations. 4. 𝗤𝗔 - Quality Assurance: Quality Assurance professionals focus on the overall quality management of the software process. Their role involves process auditing, setting quality benchmarks, and ensuring compliance with standards. QA is often more process-focused and oversees that the best practices are followed. - 𝗖𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻: QA ensures consistency in quality by establishing and maintaining quality processes. They help reduce errors and inefficiencies by implementing best practices across teams, which helps in minimizing risks and maintaining product reliability. 𝗜𝗻 𝘀𝘂𝗺𝗺𝗮𝗿𝘆: ‣ 𝗦𝗗𝗘𝗧 focus on automation and development support in testing. ‣ 𝗤𝗘 ensure quality is integrated throughout the development process. ‣ 𝗧𝗘 are responsible for manual and automated test execution. ‣ 𝗤𝗔 professionals focus on managing and optimizing quality processes. Each role plays a unique part in creating high-quality software, from writing automated tests to ensuring robust quality standards across the development lifecycle. Follow Musfiqur Rahman Foysal for more QA content.