React is known for its efficient rendering process, and one of the key techniques it utilizes is batching. Batching refers to the process of combining multiple updates into a single update, minimizing the number of times the real DOM needs to be updated. In this article, we will explore how batching works under the hood in React and its impact on performance. We will also provide examples to demonstrate the concept.

Understanding Batching

When React renders a component, it doesn’t immediately apply every change to the real DOM. Instead, it batches the updates together and performs them all at once, resulting in better performance and a smoother user experience. React achieves batching through a technique called “transaction,” which groups related operations and executes them as a single unit of work.

Example: Updating State with Batching:

Let’s consider an example where a component updates its state multiple times in quick succession. Without batching, React would update the real DOM for each state change separately, even if the actual value changed only once. This approach would be inefficient.

However, due to batching, React combines these updates into a single transaction. As a result, the real DOM is updated only once after all the state changes have been processed. This reduces unnecessary DOM updates and improves performance.

To demonstrate this, let’s modify the example code to log the current transaction state:

import React from "react";

function Example() {
    const [count, setCount] = useState(0);

    const handleClick = () => {
        console.log(
            "before ",
            React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED.ReactCurrentOwner.currentDispatcher,
        );
        setCount(count + 1);
        setCount(count + 1);
        setCount(count + 1);

        console.log(
            "after ",
            React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED.ReactCurrentOwner.currentDispatcher,
        );
    };

    return (
        <div>
            <p>Count: {count}</p>
            <button onClick={handleClick}>Increment</button>
        </div>
    );
}

In this modified example, we log the current transaction state before and after the state updates. By observing the logs, we can see that React batches the updates together into a single transaction.

Common Pitfall: Event Handlers and Batching: React’s batching behavior can sometimes lead to confusion, especially when dealing with event handlers. Consider an example where a component updates its state and logs the count immediately after the state update. Due to batching, the logged count will always be one less than the actual value at that moment.

This happens because React schedules a batch of updates to the real DOM, deferring the actual application of changes. Therefore, when the log statement is executed, the state update might not have been processed yet.

Fiber Architecture and Incremental Updates

React Fiber, introduced in React 16, is a re-implemented rendering engine that offers enhanced efficiency and flexibility. It introduces incremental updates to the virtual DOM, enabling the browser to handle user input and other events without blocking the UI.

With React Fiber, updates are broken down into smaller chunks, allowing for prioritization and interleaving of different tasks. This approach enhances responsiveness and ensures that the user interface remains interactive even during intense rendering processes.

Conclusion

Batching is a fundamental mechanism in React that optimizes UI updates for improved performance. By combining multiple updates into a single transaction, React minimizes the number of DOM manipulations and delivers a smoother user experience. Understanding batching and its impact on event handling and performance is crucial for developing efficient React applications.

In this article, we discussed the concept of batching in React, its implementation through transactions, and its benefits in terms of performance. We provided examples to illustrate how batching works and explained the significance of React Fiber’s incremental updates. By leveraging batching effectively, developers can build highly performant and responsive React applications.

Thank you for reading! ❤️

Unlocking the Power of Web Scraping for Data Collection and Analysis

React Fiber is a revolutionary feature in the React library that aims to significantly enhance the performance and rendering efficiency of React applications. At the heart of Fiber lies the reconciliation process, a powerful mechanism that efficiently updates the user interface based on changes in application state. In this blog, we will delve into the concept of Fiber trees and explore how the reconciliation process works.

Reconciliation Stack

Before the release of React 16, React employed a stack-based reconciliation algorithm known as the "stack reconciler" to compare the old and new virtual trees and update the DOM accordingly. While suitable for simpler scenarios, the stack reconciler faced challenges as applications grew in complexity.

The synchronous nature of the stack reconciler caused updates to block the main thread, leading to janky and unresponsive user interfaces, especially with large virtual trees or slower devices. Additionally, changes to the virtual tree were delayed until the reconciler completed its work, affecting overall performance and responsiveness.

The Fiber Reconciler

To address these challenges and improve React's performance, React 16 introduced the Fiber reconciler, a complete rewrite of the previous stack-based algorithm with several significant benefits.

How React Fiber Works

When a React application undergoes an update triggered by user interactions or events, the reconciliation process is initiated. Its goal is to efficiently determine the parts of the user interface that require updating and apply these changes to the real DOM.

Fiber Trees - Current and Work in Progress

The reconciliation process involves two sets of virtual DOM trees: the "current" tree and the "work in progress" tree, also known as the Fiber tree. The "current" tree represents the existing state of the virtual DOM, while the "work in progress" tree is a replica that incorporates the updates to be performed.

Render Phase

The reconciliation process begins with the render phase, where React traverses the component tree and creates fiber nodes, special data structures representing different portions of the virtual DOM.

During the render phase, React compares the fiber nodes in the "current" and "work in progress" trees to identify parts of the virtual DOM requiring updates. Leveraging a work loop, React breaks rendering work into smaller units that can be interrupted and resumed, prioritizing important updates to maintain a responsive user interface.

Commit Phase

After completing the render phase and applying the necessary updates to the "work in progress" tree, React enters the commit phase. Here, React takes the updates from the "work in progress" tree and applies them to the real DOM, making the changes visible to users.

React optimizes the performance by batching updates during the commit phase, reducing unnecessary reflows and repaints for a smoother user experience.

A New Horizon for Concurrency and Batching

React Fiber, the new reconciliation architecture introduced in React 16, not only improves rendering efficiency but also unlocks powerful features for concurrency and batching. This allows React to break free from the limitations of synchronous updates and enables the library to handle complex user interactions more intelligently.

| Concurrency and Batching

In traditional synchronous rendering, React updates the entire component tree before reflecting changes in the real DOM. This approach can lead to unnecessary processing, causing performance bottlenecks and affecting user experience. React Fiber introduces the concept of concurrency, which enables asynchronous rendering and prioritization of tasks.

With concurrency, React can pause, resume, or abort rendering work as needed, and even divide the rendering process into smaller, more manageable chunks. This fine-grained control empowers React to prioritize essential updates and respond faster to user interactions, making the user interface feel more responsive and fluid

New Functionality in React Fiber

To leverage the benefits of concurrency and batching, React introduces several new functionalities:

useDeferredValue;

The useDeferredValue hook allows developers to defer the processing of non-essential updates, optimizing performance by prioritizing high-priority work. For example, in a search input, we may want to defer the rendering of search results until the user pauses typing. This way, React can focus on more critical tasks like handling user input, ensuring a smoother experience.

import { useDeferredValue } from "react";

const Search = () => {
    const [query, setQuery] = useState("");

    // Defer the processing of non-essential updates
    const deferredQuery = useDeferredValue(query);

    const handleChange = (e) => {
        setQuery(e.target.value);
    };

    // Rest of the component logic
};

useTransition;

The useTransition hook allows developers to create smoother transitions during rendering updates. It helps to manage complex interactions by defining specific points in the rendering process where React should wait for a transition to complete before displaying the changes.

import { useTransition, useEffect, useState } from "react";

const ExpensiveComponent = () => {
    const [isPending, startTransition] = useTransition();

    // Simulate a time-consuming rendering process
    useEffect(() => {
        startTransition(() => {
            // Code to render the expensive component
        });
    }, []);

    if (isPending) {
        return <p>Loading...</p>;
    }

    // Return the fully rendered component
    return <div>Expensive Component Content</div>;
};

Suspense Component

The Suspense component is a powerful tool that allows developers to handle loading states and fallback UIs in an elegant and straightforward manner. It is especially useful when dealing with asynchronous data fetching or code-splitting.

import { Suspense } from "react";

const MyComponent = () => {
    return (
        <Suspense fallback={<p>Loading...</p>}>
            {/* Asynchronously load and render the lazy component */}
            <LazyComponent />
        </Suspense>
    );
};

Conclusion

React Fiber's reconciliation process, powered by the concept of Fiber trees, plays a pivotal role in driving React's remarkable performance. By intelligently managing virtual DOM updates and efficiently applying them to the real DOM, React ensures that user interfaces remain snappy and responsive, making React a popular and powerful choice for building modern web applications. The introduction of React Fiber marks a significant milestone in React's evolution, paving the way for even more efficient and responsive applications.

Thanks for reading ❤️

There are many different approaches for getting data from the web such as writing a custom crawler from scratch, or by using one of the many existing libraries that make it absolutely easy to create a web scraping tool in Python.

In this Python tutorial we will go through a simple example of how to scrap a website to gather news articles and create simple applications using NEWSAPI library.

Most of the libraries for gathering information are Python standard, with the exception of the Pandas and NewsApi libraries accessible via installation and free registration on the official websites.

Prerequisites

Get API key

Firstly, get the API key by registering on the News API web page. Be sure to register as an individual. This will grant the free usage of the API. Click Here and fill out the form. The API key should be in your inbox shorty.

import pandas as pd
import requests
import json

API Key Note that in order to interact with the API, it is mandatory to provide the API key.

There are two ways to do that:

1. Using API key in the request URL: Include the API key as a query parameter in the URL when making requests.

    https://newsapi.org/v2/everything?q=bitcoin&apiKey=YOUR_API_KEY
   

2. Using the NewsApiClient library: Initialize the client with your API key (requires installing the newsapi-python package).

    from newsapi import NewsApiClient
    newsapi = NewsApiClient(api_key='YOUR_API_KEY')
   

Headers

we create a dictionary to store our API key, we include it in the header of the “requests”

# Create a variable to hold the API key
headers = {'Authorization': 'cd23a40b7a0d44cb99af86eeb31869eb'

API Endpoint

News API has 2 main endpoints:

• Everything /v2/everything – search every article published by over 75,000 different sources large and small in the last 3 years. This endpoint is ideal for news analysis and article discovery.

• Top headlines /v2/top-headlines – returns breaking news headlines for countries, categories, and singular publishers. This is perfect for use with news tickers or anywhere you want to display live up-to-date news headlines.

# create 2 variables to hold the API endpoints.
everything = "https://newsapi.org/v2/everything?"
top_headlines = "https://newsapi.org/v2/top-headlines?"

API Keywords

Keywords or a phrase to search for.

the parameter ‘q’ stands for the keyword search value. The news articles matching the keyword search are returned as a response

# in our tutorial we look for news related to Apple
symbols = "apple"

API Sources

A comma-seperated string of identifiers for the news sources or blogs you want headlines from.

sources = ['abc-news', 'business-insider', 'financial-post', 'google-news',
          'reuters','nbc-news', 'techcrunch', 'the-wall-street-journal']

API sortBy

The ‘sortBy’ parameter contains the option that applies to the returned articles. Based on this value, the articles are sorted in the response

Possible options: [relevancy, popularity, publishedAt].

sorby = "popularity"

Payloads

Next step is to create different payloads that need to be sent to the API. The payload is nothing more than some information that needs to be sent to the API. The payload can include information such as news category, country, language, etc.

params= {
       'q': keywords,
       'apiKey': api_key,
       'sortBy': sorby,
       'language': 'en',
       'page': 1
      }

Requests API

Make the request using the get() method of the requests library.

Make the request using the get() method of the requests library. we’re going to pass ony 2 parameters, first the ‘url’ parameter, specify the NewsAPI endpoint, and lastly pass the payloads dictionary to the ‘params’ parameter.

response = requests.get(url=top_headlines, headers=headers, params=params)

The structure of the “requests” remains the same throughout, we can update the parameters “url” and “params” parameter in order to gather diffrent type of news.

Response

Create a viariable to hold the the “response” in dictionary format

output = response.json()

Print out the response on the console by accessing the method “json”, since it already data formated by NewsAPI library.

print(json.dumps(output, indent=4))
{
    "status": "ok",
    "totalResults": 10,
    "articles": [
        {
            "source": {
                "id": null,
                "name": "Rappler"
            },
            "author": "Rappler.com",
            "title": "Jollibee introduces new sweet chili chicken - Rappler",
            "description": "Jollibee's fried chicken is coated in a sweet-spicy chili glaze",
            "url": "https://www.rappler.com/life-and-style/food-drinks/jollibee-sweet-chili-chicken",
            "urlToImage": "https://assets2.rappler.com/2021/06/1-1623468746894.jpg",
            "publishedAt": "2021-06-12T03:39:00Z",
            "content": "Jollibee's new limited edition item can be ordered for dine-in, take-out, drive-thru, or delivery via the Jollibee Delivery app, website, hotline number, GrabFood, and foodpanda. Rappler.com"
        },
        {
            "source": {
                "id": null,
                "name": "Rappler"
            },
            "author": "Reuters",
            "title": "Hong Kong democracy activist Agnes Chow released from prison - Rappler",
            "description": "(1st UPDATE) Agnes Chow serves nearly seven months in prison for her role in an unauthorized assembly during Hong Kong's 2019 anti-government protests",
            "url": "https://www.rappler.com/world/asia-pacific/hong-kong-democracy-activist-agnes-chow-released",
            "urlToImage": "https://assets2.rappler.com/2021/06/Agnes-Chow-Wikimedia-June-12-2021-1623465814534.jpeg",
            "publishedAt": "2021-06-12T02:45:00Z",
            "content": "Hong Kong pro-democracy activist Agnes Chow was released from prison on Saturday, June 12, after serving nearly seven months for her role in an unauthorized assembly during anti-government protests i\u2026 [+18 chars]"
        },

Save Resonse to CSV

In order to create a dataframe with the result obtained, we must access the dictionary values using the “articles” key

print(output.keys())
dict_keys(['status', 'totalResults', 'articles'])

Create a variable to hold the articles information.

articles = output['articles']

Using “pandas” we create a dataframe using the method “DataFrame” passing parameter “data” with the articles variable

df = pd.DataFrame(articles)
df.head(3)

You can find my Jupyter Notebook for this on my Github.

Thanks for reading and happy web scraping everyone! ❤️

Rails has a wonderful built-in mailing system called Action Mailer, which allows us to send email from our application as long as we have a mailer model. Mailers are actually a lot like controllers: they have their own app/mailers directory, and each mailer has its own associated view as well. Each mailer also inherits from ActionMailer::Base, and just like controllers, can be generated really easily.

This tutorial explains how to configure Ruby on Rails in the development environment toward sending emails via Ruby on Rails Action Mailer integrated with the most popular email providers Gmail & SendGrid.

Let’s take it step by step, first we need to do a bit of configuration, we are going to send emails from the developement enviroment.

ActionMailer Gmail Confiruation

In order to use Action Mail with Gmail, the best solution that does not violate your other security such as 2-step certification is using App Password

• Open your Account Security Settings

• Navigate to App Password

• On the App Password page, select “Mail” app, and “Other” device, now we provide the name of ruby on Rail application. Then click “Generate”.

• Copy your App password from the yellow Box.

ActionMailer Rails Confiruation

Ruby doesn’t require any additional gems to be installed, as it’s just a simple smtp setting, we just need to configure the setup by adding the following code into our file development.rb so that the Action Mailer and mailcatcher will be setup properly.

for mote informaition refer to Action Mail Confiruation for Gmail.

To start, we’ll use Rails to generate a mailer:

Creating a Mailer

So the first thing we need to do is to create a new Mailer. To do this we can use the Rails generator:

rails generate mailer UserMailer welcome_message
create app/mailers/user_mailer.rb
invoke erb
create app/views/user_mailer
create app/views/user_mailer/welcome_message.text.erb
create app/views/user_mailer/welcome_message.html.erb

Running this command in the terminal generates a few different files for us. We now have an app/mailers directory, with an user_mailer.rb and application_mailer.rb file.

class ApplicationMailer < ActionMailer::Base
default from: "from@example.com"
layout "mailer"
end

Theapplication_mailer.rb is the base class that all other mailers are going to inhert from, we notice it inherts from ActionMailer::Base and the first thing it does it cause the default method default from: "from@example.com" and passes some options, these option will be applied to all of our email by default because it's defined up here in the base class. The from option is the email address you want to use as the sender of the email.

class UserMailer < ApplicationMailer
def welcome_message
@greeting = "Hi"
mail to: "to@example.org", subject: 'Account registration'
end
end

It also generated this UserMailer, notice that the user_mailer.rb inherts from the ApplicationMailer.

Now this mailer classes works similarlly to rails controllers, they inherts from the common base class, in this case “ApplicationMailer”. Thet also has a method inside of him, notice that this method correspond to the name we gave in the command line “welcome_message” , and those methods are similiar to controller actions, but instead of generating an HTML response, “mailer” method generate an “email”.

So this welcome_message is goning to generate the email we want. we also see that step down method includes this “@greeting” instance variable, an instance variables define inside of welcome_message method are accessabile in view templates just like we would expect with the controller action.

And the last line of these mailer methods must call this mail method to accualty create a mail message a return it. now you notice we can also pass option to this method, in this case we have the option to:, this the person we want to send the email to.

To create that email, the mail method is goning to render some email template, now the generator went ahead and created 2 email templates for us to render that new message. and those templates are over in:

App
|- Views
|- user_mail
|- welcome_message.html.erb
|- welcome_message.text.erb

notice that the name of the file includes the name of the mailer method, welcome_message.html.erb that generates an HTML email and welcome_message.text.erb that generates a plain text email.

If we take look at the HTML template, what we see it looks just like any other view template.

It’s a good practice to send both plain text and HTML email and let the email client decide which one to display, and the action mailer does all the heavy lifting for us because we have 2 views templates with the same name but different content types ebedded in their file names, both templates will get rendered and sent out in one multi-part email.

Let’s go ahead and setup a user login with our welcome_message method, so our users_controller.rb looks like:

def create
@user = User.new(signup_params)
if @user.save!
session[:user_id] = @user.id
UserMailer.with(user: current_user).welcome_msg.deliver_now
render json: @user, status: :created
end
end

It would be handy to change our template so that it generates an email content we want.

Welcome to [www.example.com,](http://www.example.com,) <%= @user.name %>

You have successfully signed up to www.example.com

Your email address is : <%= @user.email%>

Your user name is : <%= @user.user\_name%>

Thanks for joining and have a great day!

Conclusion

It doesn’t really matter if you are building a consumer or business application, you’re going to need to send emails to your users as some point.

Thank you for reading! ❤️

One of the most common patterns you’ll see in React when dealing with forms is that we need to make React aware of changes to the form inputs when the user types. This means that the component containing those input elements typically needs to keep track of the text in the input elements using state.

React supports two types of components: controlled components and uncontrolled components. Let’s uncover the main differences between controlled and uncontrolled components.

Controlled Component

Unlike the uncontrolled component, the input form element in the controlled component is handled by the component rather than the DOM. It takes its current value through props. The changes are notified through useState Hook. Let’s consider an example to understand it in a better way.

import { useState } from "react";
export default function ControlledComponent() {
    const [name, setName] = useState("");
    const [age, setAge] = useState();
    const [hairColor, setHairColor] = useState("");
    return (
        <form>
            <label htmlFor="name">
                Name
                <input
                    name="name"
                    type="text"
                    placeholder="Name"
                    value={name}
                    onChange={(e) => setName(e.target.value)}
                />
            </label>
            <label htmlFor="age">
                Age
                <input
                    name="age"
                    type="number"
                    placeholder="Age"
                    value={age}
                    onChange={(e) => setAge(Number(e.target.value))}
                />
            </label>
            <label htmlFor="haircolor">
                Hair Color
                <input
                    name="haircolor"
                    type="text"
                    placeholder="Hair Color"
                    value={hairColor}
                    onChange={(e) => setHairColor(e.target.value)}
                />
            </label>
            <button type="submit">Submit</button>
        </form>
    );
}

​

The input field is Controlled because React sets its value from the state .

<input value={value} />

Whenever we type into the input field, the onChange handler updates the state with the input’s value accessed from the event object. The value state variable is the source of truth. When we need to access the value entered by the user into the input field, we have to read the value from the state variable. The Controlled components can help access the value of any input field.

Uncontrolled component

Similar to the traditional HTML form inputs, the uncontrolled component can be written using a ref to get form values from the DOM. Thus there is no need to write an event handler for every state update and the HTML elements maintain their own state that will be updated when the input value changes.

import { createRef } from "react";
export default function UncontrolledComponent() {
    const nameInput = createRef();
    const ageInput = createRef();
    const hairColorInput = createRef();
    const handleSubmit = (e) => {
        e.preventDefault();
        console.log(nameInput.current.value);
        console.log(ageInput.current.value);
        console.log(hairColorInput.current.value);
    };
    return (
        <div>
            <form onSubmit={handleSubmit}>
                <label htmlFor="name">
                    Name
                    <input name="name" type="text" placeholder="Name" ref={nameInput} />
                </label>
                <label htmlFor="age">
                    Age
                    <input name="age" type="number" placeholder="Age" ref={ageInput} />
                </label>
                <label htmlFor="haircolor">
                    Hair Color
                    <input name="haircolor" type="text" placeholder="Hair Color" ref={hairColorInput} />
                </label>
                <button type="submit">Submit</button>
            </form>
        </div>
    );
}

Conclusion

Both the controlled and uncontrolled form fields have their merit. Evaluate your specific situation and pick the approach — what works for you is good enough.

As always, thanks for reading!

A confusion matrix is a table that is often used to describe the performance of a classification model, the result obtained from the confusion matrix is obtained by comparing the result of the test dataset for which the true values known. depending on the type of classification model the number of classes may vary but the concept is very simple even the correlated terminology can create confusion.

A confusion matrix is useful in the supervised learning category of machine learning using a labelled data set. As shown below, it is represented by a table. This is a sample confusion matrix for a binary classifier (i.e. 0-Negative or 1-Positive)

Understanding the Confusion Matrix

True Positive (TP)

The predicted value matches the actual value
The actual value was positive and the model predicted a positive value

True Negative (TN)

The predicted value matches the actual value
The actual value was negative and the model predicted a negative value

False Positive (FP) — Type 1 error

The predicted value was falsely predicted
The actual value was negative but the model predicted a positive value
Also known as the Type 1 error

False Negative (FN) — Type 2 error

The predicted value was falsely predicted
The actual value was positive but the model predicted a negative value
Also known as the Type 2 error

Classification Metrics

There are multiple accuracy metrics that can be generated from the confusion matrix.

Accuracy

Accuracy (ACC) is calculated as the number of all correct predictions divided by the total number of the dataset. The best accuracy is 1.0, whereas the worst is 0.0. It can also be calculated by 1 — ERR.

Precision

Precision is the number of True Positives divided by the number of True Positives and False Positives. Put another way, it is the number of positive predictions divided by the total number of positive class values predicted. It is also called the Positive Predictive Value (PPV)

Recall

Precision is the ratio of true positives to the total of the true positives and false positives. Precision looks to see how much junk positives got thrown in the mix. If there are no bad positives (those FPs), then the model had 100% precision. The more FPs that get into the mix, the uglier that precision is going to look.

F1 Score

F1 Score is the weighted average of Precision and Recall. Therefore, this score takes both false positives and false negatives into account. Intuitively it is not as easy to understand as accuracy, but F1 is usually more useful than accuracy, especially if you have an uneven class distribution. Accuracy works best if false positives and false negatives have similar cost. If the cost of false positives and false negatives are very different, it’s better to look at both Precision and Recall. In our case, F1 score is 0.701

Thanks for reading!

Sentiment analysis is a powerful tool that businesses can leverage to better understand the overall opinions of their customer, gain insights and make data-driven decisions.

In this tutorial we will classify text articles using VADER on a dataset from CrowdFlower.

Import Libraries

import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.probability import FreqDist
from nltk.tokenize import sent_tokenize, RegexpTokenizer
from nltk.sentiment.vader import SentimentIntensityAnalyzer
nltk.download('stopwords')
nltk.download('vader_lexicon')
nltk.download('punkt')

Data Collection

For the sake of this tutorial we’re going to use a simple dataset from CrowdFlower via data.world (IMDB Sentiment Sampled). For more info visit the link:

https://data.world/robbertb/imdb-sentiment-sampled

df = pd.read_csv('imdb_sentiment.csv')
df.head()

review
0Protocol is an implausible movie whose only sa...
1I just watched The Dresser this evening, havin...
2Besides being boring, the scenes were oppressi...
3I'm not sure why the producers needed to trade...
4Honestly - this short film sucks. the dummy us...

Preprocessing

The NLTK module is a massive tool kit, aimed to help you with the entire Natural Language Processing (NLP) methodology. NLTK will aid you with everything from splitting sentences from paragraphs, splitting up words, recognizing the part of speech of those words.

In order to apply polarity score we need to prepare your data, we start by the converting our data to lowercase using the built-in function “lower”.

df['review'] = df['review'].apply(lambda txt: txt.lower())

Removing stop word using NLTK by downloading the english stopwords.

stop_words=stopwords.words('english')
df['review'] = df['review'].apply(lambda txt: ' '.join([word for word in txt.split() if word not in stop_words]))

Sentences Tokenization

df['review'] = df['review'].apply(lambda txt: sent_tokenize(txt))

Join the tokenized data into text.

df['review'] = df['review'].apply(lambda txt: ' '.join(txt))

VADER

VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media.

It is used for sentiment analysis of text which has both the polarities i.e. positive/negative. VADER is also used to quantify how much positive or negative emotion the text has and also the intensity of emotion.

Polarity classification

The VADER library returns 4 values such as:

• pos: The probability of the sentiment to be positive

• neu: The probability of the sentiment to be neutral

• neg: The probability of the sentiment to be negative

• compound: The normalized compound score which calculates the sum of all lexicon ratings and takes values from -1 to 1

Notice that the pos, neu and neg probabilities add up to 1. Also, the compound score is a very useful metric in case we want a single measure of sentiment. Typical threshold values are the following:

• positive: compound score>=0.05

• neutral: compound score between -0.05 and 0.05

• negative: compound score<=-0.05

Instantiate a new object with NLTK SentimentIntensityAnalyzer

sid = SentimentIntensityAnalyzer()

Now we can create a new column to the original DataFrame to store the polarity_scores dictionary, the scores extracted will have the keys “neg”, “neu”, “pos”, “compound” derived from the composite score.

df['score'] = df['review'].apply(lambda txt: sid.polarity_scores(txt))

df['negative'] = df['score'].apply(lambda txt: txt['neg'])

df['neutral'] = df['score'].apply(lambda txt: txt['neu'])

df['positive'] = df['score'].apply(lambda txt: txt['pos'])

df['compound'] = df['score'].apply(lambda txt: txt['compound'])

We will follow by creating a function called “polarity_score” to calculate the accuracy test for each review in our dataframe. We can finally apply the function by creating a new column called “sentiment”.

The reviews in this column will be classified into positive, negative and neutral.

def polarity_score(compound):
if compound > 0.05:
return "positive"
elif compound < -0.5:
return "negative"
elif compound >= -0.05 and compound < 0.05:
return "neutral"

df['sentiment'] = df['compound'].apply(lambda val: polarity_score(val))
df.head()

df['sentiment'].value_counts()

Conclusion

VADER classifies the sentiments very well. It is easy to use, the ready-made model which can be used across multiple domains, social-media texts.

Thanks for reading and happy web scraping everyone!

You can find my Jupyter Notebook for this on my Github.

Introduction

Logistic regression is a classification algorithm used to assign observations to a discrete set of classes. Unlike linear regression which outputs continuous number values, logistic regression transforms its output using the logistic sigmoid function to return a probability value which can then be mapped to two or more discrete classes.

Logistic regression is used to calculate the probability of a binary event occurring and to deal with issues of classification. For example, predicting if an incoming email is spam or not spam, or predicting if a credit card transaction is fraudulent or not fraudulent. In a medical context, logistic regression may be used to predict whether a tumor is benign or malignant. In marketing, it may be used to predict if a given user (or group of users) will buy a certain product or not. An online education company might use logistic regression to predict whether a student will complete their course on time or not.

What are the different types of logistic regression?

The three types of logistic regression are:

-- Binary logistic regression is the statistical technique used to predict the relationship between the dependent variable (Y) and the independent variable (X), where the dependent variable is binary in nature. For example, the output can be Success/Failure, 0/1 , True/False, or Yes/No. This is the type of logistic regression that we’ve been focusing on in this post.

-- Multinomial logistic regression is used when you have one categorical dependent variable with two or more unordered levels (i.e two or more discrete outcomes). It is very similar to logistic regression except that here you can have more than two possible outcomes. For example, let’s imagine that you want to predict what will be the most-used transportation type in the year 2030. The transport type will be the dependent variable, with possible

-- outputs of train, bus, tram, and bike (for example). Ordinal logistic regression is used when the dependent variable (Y) is ordered (i.e., ordinal). The dependent variable has a meaningful order and more than two categories or levels. Examples of such variables might be t-shirt size (XS/S/M/L/XL), answers on an opinion poll (Agree/Disagree/Neutral), or scores on a test (Poor/Average/Good).

Advantages of logistic regression

Advantages

• Logistic regression is easier to implement, interpret, and very efficient to train.

• It makes no assumptions about distributions of classes in feature space.

• It can easily extend to multiple classes(multinomial regression) and a natural probabilistic view of class predictions.

• Logistic regression works well for cases where the dataset is linearly separable.

Disadvantages

• Logistic regression fails to predict a continuous outcome.

• If the number of observations is lesser than the number of features, Logistic Regression should not be used, otherwise it may lead to overfitting.

• Logistic Regression requires average or no multicollinearity between independent variables.

• Logistic regression may not be accurate if the sample size is too small.

What is Sigmoid Function

In order to map predicted values to probabilities, we use the sigmoid function. The function map any real input t and outputs a value between zero and one. For the logit, this is interpreted as taking input log-odds and having output probability. the logistic “sigmoid” function is defined as follow:

To understand how sigmoid function squashes the values within the range, let’s visualize the graph of the sigmoid function.

As you can see from the graph, the sigmoid function becomes asymptote to y=1 for positive values of x and becomes asymptote to y=0 for negative values of x.

Regularization in Logistic Regression

Regularization is extremely important in logistic regression modeling. Without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions.

Regularization is the most used technique to penalize complex models in machine learning, it is deployed for reducing overfitting (or, contracting generalization errors) by putting network weights small. It also enhances the performance of models for new inputs.

Penalty Term

Through biasing data points towards specific values such as very small values to zero, regularization achieves this biasing by adding a tuning parameter to strengthen those data points:

1. L1 regularization: It adds an L1 penalty that is equal to the absolute value of the magnitude of coefficient, or simply restricting the size of coefficients. For example, Lasso regression implements this method.

2. L2 Regularization: It adds an L2 penalty which is equal to the square of the magnitude of coefficients. For example, Ridge regression and SVM implement this method.

3. Elastic Net: When L1 and L2 regularization combine together, it becomes the elastic net method, it adds a hyperparameter.

If both L1 and L2 regularization work well, you might be wondering why we need both. It turns out they have different but equally useful properties, from a practical standpoint, L1 tends to shrink coefficients to zero whereas L2 tends to shrink coefficients evenly.

L1 is therefore useful for feature selection, as we can drop any variables associated with coefficients that go to zero.

L2 on the other hand, is useful when you have collinear/codependent features.

(An example pair of codependent features is gender and ispregnant since, at the current level of medical technology, only females can be ispregnant.) Codependence tends to increase coefficient variance, making coefficients unreliable/unstable, which hurts model generality. L2 reduces the variance of these estimates, which counteracts the effect of codependencies.

Conclusion

In this blog I presented the basic concept of logistic regression. I hope this blog was helpful and motivated you enough to get interested in the topic.

For a Deeper Dive, Check the Original Post 🔗

Thank you for reading! ❤️

ope in Rails?

Scopes in Rails are special methods to run SQL queries that you can build in any rails model. A scope will always return an ActiveRecord::Relation object, even if the conditional evaluates to false , whereas a class method, will return nil. we frequently run similar queries in our rails console to query our database and also understand the structure of the result we are receiving before going further with the development of our applications.

Why use Scopes? & why is it useful

A use case for scopes is often for filtering by status, date, date ranges, ordering, groupings, and more. Compared to class methods, scope methods are incredibly fast and quite easy to return the desired result in fraction of time.

Scopes help you D.R.Y out your active record calls and also keep your codes organized. Scopes make it easy to find the records you need at a particular time.

Lets start with the basic, We’ll use a sample model named Book to illustrate what scopes allow you to do.

On your homepage, you only want to show published books of course. So in your action, you would use the following code to ensure that.

Class Book < ActiveRecord::Base
scope :published, -> { where(:published => true) }
end
# This retuns ActiveRecord::Relation object

Then in our controller or views, we can return the newly scoped objects.

class BooksController < ApplicationContrpller
def index
@books = Book.published
end
end

Taking arguments.

scopes can also receive parameters, let’s see how we can create a scope to get all books written by a specific author.

class Book < ActiveRecord::Base
scope :author_name, -> (name) { where(:authorName => name) }
end

Using conditional with scopes.

The scope named available, in the code block is a typical example of how to use conditional with scopes.

class Book < ActiveRecord::Base
scope :available, -> (bool) { where("available = ?", bool) if bool.present? }
end

Chaining

Scopes are extremely powerful due to their ability to be chained together. Think of them as a chain links

class BooksController < ApplicationContrpller
def index

return all available & published books.

@books = Book.published.available
end
end

Summary

It’s always a good idea to use scopes for selecting, sorting, joining or filtering data through your database, and then you can use class methods to chain those scopes, or even write more complex logic.

This is very important to note that the scope is intended to return an ActiveRecord::Relation object which is composable with other scopes. In short scopes should be chainable.

Hope this article helped you understand the ActiveRecord Scoping method and its uses.

Thank you for reading! ❤️

First, let's visualize the structure of our React SPA todo app:

graph TD
    A[App Component] --> B[TodoList Component]
    A --> C[AddTodo Component]
    B --> D[TodoItem Component]
    D --> E[Edit Todo]
    D --> F[Delete Todo]
    D --> G[Toggle Complete]

    H[State Management] --> A
    H --> B
    H --> C
    H --> D

In this React SPA structure, we have a main App component that contains child components for different parts of our application. The key difference from the MPA version is that all of these components exist within a single page, and React manages the rendering and updating of these components based on user interactions and state changes.

Now, let's talk about data flow and state management in this React SPA:

1. State Management: In React, we typically keep the state at the highest necessary level and pass it down to child components as props. For a simple app, we might keep all state in the App component. For more complex apps, we might use a state management library like Redux or Recoil.

2. Data Storage: Unlike the MPA where data was primarily stored on the server, in an SPA we keep a copy of the data in the client-side state. We still persist data to a server via API calls, but the immediate source of truth for the UI is the client-side state.

3. CRUD Operations: These are typically handled by updating the client-side state and making asynchronous API calls to sync with the server.

Let's walk through a typical workflow:

1. Initial Load:

   • React app loads and mounts the App component

   • App component fetches todos from the server (e.g., via a REST API)

   • State is updated with the fetched todos, causing a re-render

2. Add Todo:

   • User inputs new todo in AddTodo component

   • On submit, state is updated with new todo

   • An API call is made to add the todo on the server

3. Edit Todo:

   • User clicks edit on a TodoItem

   • TodoItem switches to edit mode (managed by local state)

   • On save, parent state is updated and an API call is made

4. Delete Todo:

   • User clicks delete on a TodoItem

   • Todo is removed from state, causing an immediate UI update

   • An API call is made to delete the todo on the server

5. Toggle Complete:

   • User clicks to toggle todo status

   • State is immediately updated, reflecting the change in the UI

   • An API call is made to update the status on the server

Now, let's look at a simplified code example for our main App component:

import React, { useState, useEffect } from "react";
import TodoList from "./TodoList";
import AddTodo from "./AddTodo";
import { fetchTodos, addTodo, updateTodo, deleteTodo } from "./api";

function App() {
    const [todos, setTodos] = useState([]);

    useEffect(() => {
        fetchTodos().then(setTodos);
    }, []);

    const handleAddTodo = (title) => {
        const newTodo = { id: Date.now(), title, completed: false };
        setTodos([...todos, newTodo]);
        addTodo(newTodo);
    };

    const handleToggle = (id) => {
        setTodos(todos.map((todo) => (todo.id === id ? { ...todo, completed: !todo.completed } : todo)));
        const todoToUpdate = todos.find((todo) => todo.id === id);
        updateTodo({ ...todoToUpdate, completed: !todoToUpdate.completed });
    };

    const handleDelete = (id) => {
        setTodos(todos.filter((todo) => todo.id !== id));
        deleteTodo(id);
    };

    return (
        <div>
            <h1>My Todos</h1>
            <AddTodo onAdd={handleAddTodo} />
            <TodoList todos={todos} onToggle={handleToggle} onDelete={handleDelete} />
        </div>
    );
}

export default App;

In this React example:

1. We use the useState hook to manage our todos state.

2. useEffect is used to fetch todos when the component mounts.

3. CRUD operations are handled by functions that update the state and make API calls.

4. The UI automatically updates when the state changes, thanks to React's reactivity.

The key differences from the MPA approach are:

1. Single Page: Everything happens within a single page load. React handles updating the UI.

2. Client-Side State: The immediate source of truth is the client-side state, not the server.

3. Asynchronous Updates: API calls happen asynchronously, allowing for immediate UI updates.

4. Component Structure: The UI is broken down into reusable components.

This approach provides a more fluid user experience, as most interactions don't require a full page reload. However, it also means we need to carefully manage state and keep it in sync with the server.

Is this helpful in understanding the SPA approach? Would you like me to elaborate on any specific part of this workflow or dive deeper into how React manages updates?

Several measures of variability exist, each with distinct strengths and weaknesses. Commonly used ones include:

1. Range: The simplest measure, calculated by subtracting the smallest from the largest value in the dataset.

2. Interquartile Range (IQR): A more robust measure less sensitive to outliers, obtained by subtracting the first quartile from the third quartile.

3. Variance: Indicates how much data deviates from the mean, calculated by summing squared differences between each data point and the mean.

4. Standard Deviation: The square root of variance, a widely used measure conveying average deviation from the mean.

Understanding variability is pivotal for making sense of data, enabling insights into dataset characteristics for more informed decision-making.

What is Coefficient of Variation (CV)?

Variability assessment is crucial in statistics, referring to the degree of dispersion or spread in a dataset. The Coefficient of Variation (CV) emerges as one such measure, representing the ratio of standard deviation to the mean. Often expressed as a percentage, CV facilitates comparing variation between different datasets. Widely applied in finance, biology, and engineering, CV aids in gauging risk or uncertainty within datasets.

Key Points about Coefficient of Variation:

1. Formula: CV = (Standard Deviation / Mean) x 100%. It gauges the relative variability of a dataset, expressed as a percentage.

2. Comparative Analysis: Used to compare variability across datasets with different units, aiding decisions in various fields.

3. Risk Assessment: Valuable in finance for evaluating investment risk. Higher CV suggests higher risk and volatility.

4. Reliability Assessment: Applied to assess the reliability of measurement tools, such as blood glucose monitors.

5. Interpretation: A lower CV signifies a more consistent dataset, while a higher CV implies greater variability. Interpretation context varies with the dataset and field of study.

Calculating Coefficient of Variation

In measuring variability, the Coefficient of Variation (CV) stands out as a versatile statistical tool widely employed in finance, biology, and engineering. As a relative measure, CV represents standard deviation as a percentage of the mean, offering a robust means to compare datasets with different units or means. Let's delve into how to calculate CV, providing insights into its interpretation and application.

1. Formula for Calculating CV:

   • CV = (Standard Deviation / Mean) x 100%

Example: Consider daily stock returns over ten days: 1.2%, -0.5%, 0.8%, -1.1%, 2.3%, 0.9%, -2.0%, -0.7%, 1.5%, 0.3%. Mean return = 0.34%, Standard Deviation = 1.48%, CV = (1.48 / 0.34) x 100% = 435.29%.

2. Interpretation of CV:

   • Low CV (< 15%): Low variability.

   • Moderate CV (15% - 30%): Moderate variability.

   • High CV (> 30%): High variability.

Interpretation varies by field; e.g., in finance, CV < 30% may be considered low.

3. Application of CV:

   • Finance: Measures risk and volatility.

   • Biology: Gauges variability in biological traits.

   • Engineering: Assesses manufacturing process variability.

   • Medicine: Evaluates variability in medical test results.

4. Interpreting Coefficient of Variation

Coefficient of Variation

The Coefficient of Variation (CV) serves as a dimensionless metric to measure the relative variability of data. Its interpretation is pivotal, offering insights into data consistency and reliability. Widely used in finance, engineering, and biology, understanding how to interpret CV is crucial for informed decision-making.

Ways to Interpret CV:

1. Low CV: Indicates tightly clustered data points around the mean, suggesting consistency and reliability. Useful in contexts like test score reliability.

2. High CV: Signifies widely dispersed data points, indicating lower consistency and reliability. Relevant in scenarios such as salary distribution analysis.

3. Comparing CVs: Facilitates insights into relative variability between datasets. Useful in identifying higher variability in one group over another.

Interpreting CV is essential for understanding

Thank you for reading! ❤️