Tools in Data Science

Mon, 01 Jan 0001 00:00:00 +0000

Actor Network Visualization#

Find the shortest path between Govinda & Angelina Jolie using IMDb data using Python: networkx or scikit-network .

You’ll learn how to analyze and visualize the social network of actors using IMDb data, covering:

Data Acquisition and Preparation: Learn how to download and work with IMDb’s non-commercial datasets. This includes using Python to read and process large TSV files.
Data Filtering and Wrangling: Understand how to clean and filter large datasets to focus on relevant information, such as specific actor categories (actors/actresses) and movie titles, to make the data more manageable for analysis.
Actor Pairing and Network Creation: Discover how to identify significant collaborations between actors by setting thresholds for the minimum number of films and co-appearances. You will learn to create a network of actors based on these pairings.
Handling Data Inconsistencies: Learn techniques to manage real-world data issues, such as variations in actor names, to ensure the accuracy of your network.
Network Analysis with Python: Get introduced to powerful Python libraries for network analysis like networkx and scikit-network to build, manipulate, and study the structure of complex networks.
Finding the Shortest Path: Learn how to apply graph algorithms to find the shortest path between two actors in the network, demonstrating the “degrees of separation” concept.
Jupyter Notebook for Data Storytelling: See how to use Jupyter Notebooks to document and present a data analysis project, combining code, text, and visualizations to tell a compelling story.
Exporting Results: Learn how to export your findings, such as the actor pairings, into formats like Excel for further use or presentation.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding in the CLI#

Command-line AI coding tools bring the power of large language models directly to your terminal, enabling scripted automation, pipeline integration, and efficient developer workflows. These tools excel at batch processing, git integration, and system-level automation that GUI tools can’t match.

In this module, you’ll learn:

Claude Code CLI: Terminal-first coding agent for interactive development and review
Codex CLI (OpenAI): Local coding agent with sandbox, approvals, CI-friendly exec
GitHub Copilot CLI: GitHub-native terminal agent with policy-aware approvals and MCP extensions
Gemini CLI: Large-context, multimodal agent with non-interactive mode
Simon Willison’s llm: Shell-native AI with UNIX pipeline integration and extensive tooling

Claude Code CLI#

Claude Code is Anthropic’s terminal-first coding agent (npm: @anthropic-ai/claude-code). It runs locally in your project, understands your codebase, and assists with routine tasks, explanations, and git workflows.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Context Engineering#

Context engineering is the systematic approach to providing AI coding assistants with the right information at the right level of detail. Unlike general prompt engineering, code context engineering focuses specifically on code-related workflows, project structure, and development processes.

Effective context engineering transforms AI from a simple code generator into an intelligent development partner that understands your project’s architecture, conventions, and goals.

In this module, you’ll learn:

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding in IDEs#

AI-powered coding assistants have changed how we write code within traditional IDEs and editors. These tools provide intelligent code completion, chat interfaces, and automated refactoring capabilities that dramatically increase developer productivity.

In this module, you’ll learn:

Copy-paste workflows: Fast iteration cycles using web-based AI assistants for quick fixes and code generation
GitHub Copilot: The industry-standard AI pair programmer for VS Code and other editors
Cursor: An AI-native editor with advanced context understanding and agent capabilities
Windsurf: The first agentic IDE with deep codebase awareness and collaborative AI features

Copy-paste workflows#

The simplest and often most effective AI coding workflow involves copying code and errors to web-based AI assistants, then pasting the improved solutions back into your editor. This approach works with any editor and provides access to the latest AI models.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Online#

Modern AI coding tools have transformed how we build software, enabling rapid prototyping and deployment through natural language interfaces. This module covers the mindset shift toward “vibe-coding” and the essential tools for AI-assisted development.

In this module, you’ll learn:

Inline AI coding sandboxes: Use Claude Artifacts, Gemini Canvas, and ChatGPT Canvas for iterative development
Hosted agent workbenches: Leverage Replit Agent, Bolt.new, and Lovable for end-to-end app creation

Artifacts: Inline AI coding sandboxes#

AI artifacts provide side-by-side code and chat interfaces that revolutionize how we iterate on applications. These tools show visible diffs, run code in real-time, and allow for conversational editing.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Strategies#

Strategic approaches to AI-assisted software development have evolved rapidly in 2025. Unlike simple code completion, these methodologies focus on sustainable, team-oriented workflows that maximize AI productivity while maintaining code quality and developer control.

Modern AI coding strategies combine human expertise with AI capabilities through structured patterns, parallel processing, and iterative feedback loops. The most effective approaches treat AI as a collaborative partner rather than a replacement, establishing clear boundaries and verification processes.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Tests#

Testing AI-generated code is about establishing fast, trustworthy feedback loops before the code lands in main. The goal is not perfect certainty—it is a reliable signal that the patch does what it should, fails when it must, and keeps shipping velocity high.

In this module, you’ll learn:

Scorecard-driven evaluation loops: Track AI agent success with automated suites and selectors
Premortem test-first flows: Lock in failing tests before asking an agent to code
Python-first fuzzing and mutation: Catch almost-right code with Hypothesis and mutmut
Agent-powered smoke tests: Let Playwright bots guard critical user journeys

Build evaluation loops#

Benchmarks stop vibe-coded fixes from quietly regressing. Start with a thin scorecard that runs on every candidate patch, then expand.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Tools#

AI coding agents perform best when you stage a predictable toolbox around them. The goal is not to spray every binary into the sandbox, but to focus on fast, auditable utilities that agents can compose for data prep, testing, review, and automation. Keep the happy path linear: expose tools, document how to call them, and wire safe defaults before inviting an agent into the repo.

In this module, you’ll learn:

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding#

AI-assisted development pairs human judgment with generative models across every workspace. This module shows how to set up productive collaborations, feed models the right project context, and ship code you trust.

You’ll explore:

Creative prompting that shapes tone and flow through vibe coding sessions and scenario planning.
Environment-specific tooling for working in browser editors, IDE extensions, and CLI companions without losing momentum.
Context engineering practices that stage repositories, snippets, and docs so models stay grounded in your codebase.
Quality safeguards that weave in automated tests, structured reviews, and targeted tool calls to keep AI-written code production ready.

By the end you will know when to lean on AI, how to steer it with precise context, and the guardrails that turn generated snippets into maintainable features.

Mon, 01 Jan 0001 00:00:00 +0000

Archived content#

Videos#

IITM BS Degree - Diploma Level Orientation#

Tools in Data Science Orientation#

Tools in Data Science Live Sessions#

Tools in Data Science Playlist#

Tools in Data Science Course Playlist, 66 videos

Optional: Parse & clean PDF files with Tabula#

Mon, 01 Jan 0001 00:00:00 +0000

Base 64 Encoding#

Base64 is a method to convert binary data into ASCII text. It’s essential when you need to transmit binary data through text-only channels or embed binary content in text formats.

Watch this quick explanation of how Base64 works (3 min):

Here’s how it works:

It takes 3 bytes (24 bits) and converts them into 4 ASCII characters
… using 64 characters: A-Z, a-z, 0-9, + and / (padding with = to make the length a multiple of 4)
There’s a URL-safe variant of Base64 that replaces + and / with - and _ to avoid issues in URLs
Base64 adds ~33% overhead (since every 3 bytes becomes 4 characters)

Common Python operations with Base64:

Mon, 01 Jan 0001 00:00:00 +0000

Terminal: Bash#

UNIX shells are the de facto standard in the data science world and Bash is the most popular. This is available by default on Mac and Linux.

On Windows, install Git Bash or WSL to get a UNIX shell.

Watch this video to install WSL (12 min).

Watch this video to understand the basics of Bash and UNIX shell commands (75 min).

Mon, 01 Jan 0001 00:00:00 +0000

BBC Weather location ID with Python#

You’ll learn how to get the location ID of any city from the BBC Weather API – as a precursor to scraping weather data – covering:

Understanding API Calls: Learn how backend API calls work when searching for a city on the BBC weather website.
Inspecting Web Interactions: Use the browser’s inspect element feature to track API calls and understand the network activity.
Extracting Location IDs: Identify and extract the location ID from the API response using Python.
Using Python Libraries: Import and use requests, json, and urlencode libraries to make API calls and process responses.
Constructing API URLs: Create structured API URLs dynamically with constant prefixes and query parameters using urlencode.
Building Functions: Develop a Python function that accepts a city name, constructs the API call, and returns the location ID.

To open the browser Developer Tools on Chrome, Edge, or Firefox, you can:

Mon, 01 Jan 0001 00:00:00 +0000

Cleaning Data with OpenRefine#

This session covers the use of OpenRefine for data cleaning, focusing on resolving entity discrepancies:

Data Upload and Project Creation: Import data into OpenRefine and create a new project for analysis.
Faceting Data: Use text facets to group similar entries and identify frequency of address crumbs.
Clustering Methodology: Apply clustering algorithms to merge similar entries with minor differences, such as punctuation.
Manual and Automated Clustering: Learn to merge clusters manually or in one go, trusting the system’s clustering accuracy.
Entity Resolution: Clean and save the data by resolving multiple versions of the same entity using Open Refine.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Notebooks: Google Colab#

Google Colab is a free, cloud-based Jupyter notebook environment that’s become indispensable for data scientists and ML practitioners. It’s particularly valuable because it provides free access to GPUs and TPUs, and for easy sharing of code and execution results.

While Colab is excellent for prototyping and learning, its free tier has limitations - notebooks time out after 12 hours, and GPU access can be inconsistent.

Learn how to mount Google Drive for persistent storage, manage dependencies with !pip install commands, as these are common pain points when getting started.

Mon, 01 Jan 0001 00:00:00 +0000

Converting HTML to Markdown#

When working with web content, converting HTML files to plain text or Markdown is a common requirement for content extraction, analysis, and preservation. For example:

Content analysis: Extract clean text from HTML for natural language processing
Data mining: Strip formatting to focus on the actual content
Offline reading: Convert web pages to readable formats for e-readers or offline consumption
Content migration: Move content between different CMS platforms
SEO analysis: Extract headings, content structure, and text for optimization
Archive creation: Store web content in more compact, preservation-friendly formats
Accessibility: Convert content to formats that work better with screen readers

This tutorial covers both converting existing HTML files and combining web crawling with HTML-to-text conversion in a single workflow – all using the command line.

Mon, 01 Jan 0001 00:00:00 +0000

Converting PDFs to Markdown#

PDF documents are ubiquitous in academic, business, and technical contexts, but extracting and repurposing their content can be challenging. This tutorial explores various command-line tools for converting PDFs to Markdown format, with a focus on preserving structure and formatting suitable for different use cases, including preparation for Large Language Models (LLMs).

Use Cases:

LLM training and fine-tuning: Create clean text data from PDFs for AI model training
Knowledge base creation: Transform PDFs into searchable, editable markdown documents
Content repurposing: Convert academic papers and reports for web publication
Data extraction: Pull structured content from PDF documents for analysis
Accessibility: Convert PDFs to more accessible formats for screen readers
Citation and reference management: Extract bibliographic information from academic papers
Documentation conversion: Transform technical PDFs into maintainable documentation

PyMuPDF4LLM#

PyMuPDF4LLM is a specialized component of the PyMuPDF library that generates Markdown specifically formatted for Large Language Models. It produces high-quality markdown with good preservation of document structure. It’s specifically optimized for producing text that works well with LLMs, removing irrelevant formatting while preserving semantic structure. Requires PyTorch, which adds dependencies but enables more advanced processing capabilities.

Mon, 01 Jan 0001 00:00:00 +0000

Correlation with Excel#

You’ll learn to calculate and interpret correlations using Excel, covering:

Enabling the Data Analysis Tool Pack: Steps to enable the Excel data analysis tool pack.
Correlation Analysis: Understanding statistical association between variables.
Creating a Correlation Matrix: Steps to generate and interpret a correlation matrix.
Scatterplots and Trendlines: Plotting data and adding trend lines to visualize correlations.
Analyzing Results: Comparing correlation coefficients and understanding their implications.
Insights and Further Analysis: Interpreting scatterplots and planning further analysis for deeper insights.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

CORS (Cross-Origin Resource Sharing) is a security mechanism that controls how web browsers handle requests between different origins (domains, protocols, or ports). Data scientists need CORS for APIs serving data or analysis to a browser on a different domain.

Watch this practical explanation of CORS (3 min):

Key CORS concepts:

Same-Origin Policy: Browsers block requests between different origins by default
CORS Headers: Server responses must include specific headers to allow cross-origin requests
Preflight Requests: Browsers send OPTIONS requests to check if the actual request is allowed
Credentials: Special handling required for requests with cookies or authentication

If you’re exposing your API with a GET request publicly, the only thing you need to do is set the HTTP header Access-Control-Allow-Origin: *.

Mon, 01 Jan 0001 00:00:00 +0000

Crawling with the CLI#

Since websites are a common source of data, we often download entire websites (crawling) and then process them offline.

Web crawling is essential in many data-driven scenarios:

Data mining and analysis: Gathering structured data from multiple pages for market research, competitive analysis, or academic research
Content archiving: Creating offline copies of websites for preservation or backup purposes
SEO analysis: Analyzing site structure, metadata, and content to improve search rankings
Legal compliance: Capturing website content for regulatory or compliance documentation
Website migration: Creating a complete copy before moving to a new platform or design
Offline access: Downloading educational resources, documentation, or reference materials for use without internet connection

The most commonly used tool for fetching websites is wget . It is pre-installed in many UNIX distributions and easy to install.

Mon, 01 Jan 0001 00:00:00 +0000

CSS Selectors#

CSS selectors are patterns used to select and style HTML elements on a web page. They are fundamental to web development and data scraping, allowing you to precisely target elements for styling or extraction.

For data scientists, understanding CSS selectors is crucial when:

Web scraping with tools like Beautiful Soup or Scrapy
Selecting elements for browser automation with Selenium
Styling data visualizations and web applications
Debugging website issues using browser DevTools

Watch this comprehensive introduction to CSS selectors (20 min):

Mon, 01 Jan 0001 00:00:00 +0000

Data Aggregation in Excel#

You’ll learn data aggregation and visualization techniques in Excel, covering:

Data Cleanup: Remove empty columns and rows with missing values.
Creating Excel Tables: Convert raw data into tables for easier manipulation and formula application.
Date Manipulation: Extract week, month, and year from date columns using Excel functions (WEEKNUM, TEXT).
Color Scales: Apply color scales to visualize clusters and trends in data over time.
Pivot Tables: Create pivot tables to aggregate data by location and date, summarizing values weekly and monthly.
Sparklines: Use sparklines to visualize trends within pivot tables, making data patterns more apparent.
Data Bars: Implement data bars for graphical illustrations of numerical columns, showing trends and waves.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Analysis with Datasette#

Datasette is an open-source tool for exploring and publishing data. Created by Simon Willison, it turns any SQLite database into an instant web interface with powerful exploration features, JSON APIs, and visualization capabilities—all without writing code.

Unlike traditional database tools that require SQL knowledge upfront, Datasette provides an interactive interface for exploring data through faceting, filtering, and full-text search. It’s particularly powerful for data journalism, analysis workflows, and sharing datasets with non-technical audiences.

Mon, 01 Jan 0001 00:00:00 +0000

Data Analysis with DuckDB#

You’ll learn how to perform data analysis using DuckDB and Pandas, covering:

Parquet for Data Storage: Understand why Parquet is a faster, more compact, and better-typed storage format compared to CSV, JSON, and SQLite.
DuckDB Setup: Learn how to install and set up DuckDB, along with integrating it into a Jupyter notebook environment.
File Format Comparisons: Compare file formats by speed and size, observing the performance difference between saving and loading data in CSV, JSON, SQLite, and Parquet.
Faster Queries with DuckDB: Learn how DuckDB uses parallel processing, columnar storage, and on-disk operations to outperform Pandas in speed and memory efficiency.
SQL Query Execution in DuckDB: Run SQL queries directly on Parquet files and Pandas DataFrames to compute metrics such as the number of unique flight routes delayed by certain time intervals.
Memory Efficiency: Understand how DuckDB performs analytics without loading entire datasets into memory, making it highly efficient for large-scale data analysis.
Mixing DuckDB and Pandas: Learn to interleave DuckDB and Pandas operations, leveraging the strengths of both tools to perform complex queries like correlations and aggregations.
Ranking and Filtering Data: Use SQL and Pandas to rank arrival delays by distance and extract key insights, such as the earliest flight arrival for each route.
Joining Data: Create a cost analysis by joining datasets and calculating total costs of flight delays, demonstrating DuckDB’s speed in joining and aggregating large datasets.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Analysis with Python#

You’ll learn practical data analysis techniques in Python using Pandas, covering:

Reading Parquet Files: Utilize Pandas to read Parquet file formats for efficient data handling.
Dataframe Inspection: Methods to preview and understand the structure of a dataset.
Pivot Tables: Creating and interpreting pivot tables to summarize data.
Percentage Calculations: Normalize pivot table values to percentages for better insights.
Correlation Analysis: Calculate and interpret correlation between variables, including significance testing.
Statistical Significance: Use statistical tests to determine the significance of observed correlations.
Datetime Handling: Extract and manipulate date and time information from datetime columns.
Data Visualization: Generate and customize heat maps to visualize data patterns effectively.
Leveraging AI: Use ChatGPT to generate and refine analytical code, enhancing productivity and accuracy.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Analysis with SQL#

You’ll learn how to perform data analysis using SQL (via Python), covering:

Database Connection: How to connect to a MySQL database using SQLAlchemy and Pandas.
SQL Queries: Execute SQL queries directly from a Python environment to retrieve and analyze data.
Counting Rows: Use SQL to count the number of rows in a table.
User Activity Analysis: Query and identify top users by post count.
Post Concentration: Determine if a small percentage of users contribute the majority of posts using SQL aggregation.
Correlation Calculation: Calculate the Pearson correlation coefficient between user attributes such as age and reputation.
Regression Analysis: Compute the regression slope to understand the relationship between views and reputation.
Handling Large Data: Perform calculations on large datasets by fetching aggregated values from the database rather than entire datasets.
Statistical Analysis in SQL: Use SQL as a tool for statistical analysis, demonstrating its power beyond simple data retrieval.
Leveraging AI: Use ChatGPT to generate SQL queries and Python code, enhancing productivity and accuracy.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data analysis#

Data Analysis: Introduction Podcast by NotebookLM

Mon, 01 Jan 0001 00:00:00 +0000

Data Cleansing in Excel#

You’ll learn basic but essential data cleaning techniques in Excel, covering:

Find and Replace: Use Ctrl+H to replace or remove specific terms (e.g., removing “[more]” from country names).
Changing Data Formats: Convert columns from general to numerical format.
Removing Extra Spaces: Use the TRIM function to clean up unnecessary spaces in text.
Identifying and Removing Blank Cells: Highlight and delete entire rows with blank cells using the “Go To Special” function.
Removing Duplicates: Use the “Remove Duplicates” feature to eliminate duplicate entries, demonstrated with country names.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation in DuckDB#

DuckDB’s SQL engine can handle large files quickly. Below are common cleaning tasks using the DuckDB CLI.

Create a Sample Dataset#

Let’s create a sample dataset that mimics real business data patterns - incomplete customer records, time-series orders, and regional variations. Before working with messy production data, you need a controlled environment to test data cleaning techniques. This sample represents common e-commerce scenarios: missing customer info (20% of orders), seasonal patterns (15-day cycles), and geographic segmentation that drive business decisions like inventory placement and marketing campaigns.

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation in the Editor#

You’ll learn how to use a text editor Visual Studio Code to process and clean data, covering:

Format JSON files
Find all and multiple cursors to extract specific fields
Sort lines
Delete duplicate lines
Replace text with multiple cursors

Here are the links used in the video:

City-wise product sales JSON

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation in the Shell#

You’ll learn how to use UNIX tools to process and clean data, covering:

curl (or wget) to fetch data from websites.
gzip (or xz) to compress and decompress files.
wc to count lines, words, and characters in text.
head and tail to get the start and end of files.
cut to extract specific columns from text.
uniq to de-duplicate lines.
sort to sort lines.
grep to filter lines containing specific text.
sed to search and replace text.
awk for more complex text processing.

Data preparation in the shell - Notebook

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation#

Data preparation is crucial because raw data is rarely perfect.

It often contains errors, inconsistencies, or missing values. For example, marks data may have ‘NA’ or ‘absent’ for non-attendees, which you need to handle.

This section teaches you how to clean up data, convert it to different formats, aggregate it if required, and get a feel for the data before you analyze.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Sourcing#

Before you do any kind of data science, you obviously have to get the data to be able to analyze it, visualize it, narrate it, and deploy it. And what we are going to cover in this module is how you get the data.

There are three ways you can get the data.

The first is you can download the data. Either somebody gives you the data and says download it from here, or you are asked to download it from the internet because it’s a public data source. But that’s the first way—you download the data.
The second way is you can query it from somewhere. It may be on a database. It may be available through an API. It may be available through a library. But these are ways in which you can selectively query parts of the data and stitch it together.
The third way is you have to scrape it. It’s not directly available in a convenient form that you can query or download. But it is, in fact, on a web page. It’s available on a PDF file. It’s available in a Word document. It’s available on an Excel file. It’s kind of structured, but you will have to figure out that structure and extract it from there.

In this module, we will be looking at the tools that will help you either download from a data source or query from an API or from a database or from a library. And finally, how you can scrape from different sources.

Mon, 01 Jan 0001 00:00:00 +0000

Data Storytelling with LLMs#

Large Language Models (LLMs) can help create compelling data stories by assisting at every step of the data-to-story value chain:

Data Engineering (scraping, cleaning)
Data Analysis (modeling, insights)
Data Visualization (charts, narratives)

Watch this talk (30m) on data storytelling with LLMs

Prerequisites#

To follow this tutorial, you’ll need:

ChatGPT Plus ($20/month) - recommended for better models
Basic understanding of data analysis concepts

Optional but useful:

Mon, 01 Jan 0001 00:00:00 +0000

Data Storytelling#

Transcript#

Once you’ve analyzed your data and put it together into charts, you still want to communicate it in a way that people will remember, that people will share. And that’s where storytelling comes in. What we’re going to look at in this module are ways of telling stories.

There are many ways in which you can narrate a story, and one way is just to show the numbers. This in itself can be powerful. What you see here, for example, is a correlation matrix which has conditional formatting on Excel that will help you understand how two securities are correlated. And it’s pretty easy to see that, for example, the Pakistani rupee is negatively correlated with many of the securities. Just the numbers can give people an understanding of what’s happening behind the scenes and convey a story.

Mon, 01 Jan 0001 00:00:00 +0000

Data Transformation in Excel#

You’ll learn data transformation techniques in Excel, covering:

Calculating Ratios: Compute metro area to city area and metro population to city population ratios.
Using Pivot Tables: Create pivot tables to aggregate data and identify outliers.
Filtering Data: Apply filters in pivot tables to analyze specific subsets of data.
Counting Data Occurrences: Use pivot tables to count the frequency of specific entries.
Creating Charts: Generate charts from pivot table data to visualize distributions and outliers.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Visualization with ChatGPT#

ChatGPT and other Large Language Models (LLMs) can help create compelling data visualizations by:

Finding and analyzing datasets
Generating visualization code
Improving visual design
Creating data stories

Watch this workshop (2h) on creating data visualizations with ChatGPT

Prerequisites#

To follow this tutorial, you’ll need:

Gemini (free) - good for processing images and video
A ChatGPT Plus subscription ($20/month) - recommended for access to advanced models and coding capabilities
GitHub account - for publishing visualizations
Basic familiarity with HTML/CSS/JavaScript

Other useful but optional tools include:

Mon, 01 Jan 0001 00:00:00 +0000

Data Visualization with Seaborn#

Seaborn is a data visualization library for Python. It’s based on Matplotlib but a bit easier to use, and a bit prettier.

This video tutorial provides a comprehensive guide to Seaborn, a powerful data visualization library built on Matplotlib. You’ll learn how to:

Understand Seaborn’s core purpose: It’s a data visualization library built on Matplotlib that simplifies plotting, often creating complex plots with just one line of code.
Install and set up Seaborn: Learn how to install it using pip or conda, and how to import necessary libraries like NumPy, Pandas, and Matplotlib for seamless integration.
Load and manage data: Primarily use Seaborn’s built-in datasets (e.g., car_crashes, tips, flights, iris, attention) for practice, and understand how to load other file types via Pandas.
Create various distribution plots:
- Distribution plots (displot): Visualize univariate distributions (distributions of a single variable), including histograms and Kernel Density Estimation (KDE) plots, and learn to define bins.
- Joint plots (jointplot): Compare two distributions by default as a scatter plot, and generate regression lines or show KDE or hexagon distributions.
- KDE plots (kdeplot): Create standalone plots for kernel density estimations.
- Pair plots (pairplot): Plot relationships across all numerical values in a DataFrame, showing histograms on the diagonal and scatter plots elsewhere.
- Rug plots (rugplot): Visualize single column data points as sticks, indicating where data is denser.
Master various categorical plots:
- Bar plots (barplot): Analyze distributions of categorical data against numerical data, understanding variance bars and how to change the aggregation estimator (e.g., median, std, cov) using NumPy functions.
- Count plots (countplot): Simply count the number of occurrences for categorical variables.
- Box plots (boxplot): Compare variables by showing quartiles, median, standard deviation, whiskers, and outliers.
- Violin plots (violinplot): A combination of box plots and KDE plots, visualizing the density estimation of data points and how to split them for comparison.
- Strip plots (stripplot): Draw scatter plots with one categorical variable, often used with box plots, and learn to use jitter to spread points and dodge to separate categories.
- Swarm plots (swarmplot): Similar to strip plots but adjusts points to prevent overlap, often layered on top of violin plots.
Generate matrix plots for correlation and data patterns:
- Heat maps (heatmap): Visualize data in a matrix format, requiring data to be prepared using correlation matrices (.corr()) or pivot tables (.pivot_table()), and how to add annotations.
- Cluster maps (clustermap): Create hierarchically clustered heat maps that calculate distances and reposition data to find specific patterns and clusters.
Utilize powerful grid systems for complex visualizations:
- Pair grids (PairGrid): Gain specific control over plot placement within a grid, allowing you to map different plot types (e.g., scatter, histogram, KDE) to the upper, lower, or diagonal sections.
- Facet grids (FacetGrid): Print multiple plots in a grid, defining columns and rows based on categorical data, and apply various styling options to individual subplots.
Create and customize regression plots:
- Regression plots (lmplot): Study relationships between numerical variables, customize markers, sizes, and colors, and separate data into columns or rows based on other variables for multi-faceted analysis.
Apply extensive styling and customization options:
- Set overall plot styles (sns.set_style) like white, darkgrid, whitegrid, dark, and ticks.
- Adjust plot size (plt.figure(figsize=...)).
- Change plot context (sns.set_context) for different uses: paper (Jupyter), talk (presentation), poster.
- Control font scale (font_scale).
- Manage axis visibility by turning spines on or off.
- Change plot color schemes using palettes and exploring Matplotlib’s color maps.
- Reposition plot legends for better readability.
- Customize marker symbols, sizes, line widths, and edge colors in various plot types.

Mon, 01 Jan 0001 00:00:00 +0000

Data visualization#

Effective visuals connect analysis to decisions. This module spotlights the tools that help you design narratives, build interactive experiences, and adapt outputs for every kind of audience.

You’ll practice:

Story-first planning with data storytelling frameworks and LLM partners such as ChatGPT to refine messaging.
Portable presentations spanning HTML decks in RevealJS, Markdown slides in Marp, and notebook-native sessions in Marimo.
Specialized builders including RAWgraphs, Seaborn, Excel forecasting charts, Flourish animations, PowerPoint motion, and Kumu network diagrams.
Collaborative perspectives through actor network visualizations and AI-assisted storytelling that keep stakeholders aligned.

Use these resources to turn complex analyses into clear, compelling stories wherever your viewers meet them.

Mon, 01 Jan 0001 00:00:00 +0000

Data Transformation with dbt#

You’ll learn how to transform data using dbt (data build tool), covering:

dbt Fundamentals: Understand what dbt is and how it brings software engineering practices to data transformation
Project Setup: Learn how to initialize a dbt project, configure your warehouse connection, and structure your models
Models and Materialization: Create your first dbt models and understand different materialization strategies (view, table, incremental)
Testing and Documentation: Implement data quality tests and auto-generate documentation for your data models
Jinja Templating: Use Jinja for dynamic SQL generation, making your transformations more maintainable and reusable
References and Dependencies: Learn how to reference other models and manage model dependencies
Sources and Seeds: Configure source data connections and manage static reference data
Macros and Packages: Create reusable macros and leverage community packages to extend functionality
Incremental Models: Optimize performance by only processing new or changed data
Deployment and Orchestration: Set up dbt Cloud or integrate with Airflow for production deployment

Here’s a minimal dbt model example, models/staging/stg_customers.sql:

Mon, 01 Jan 0001 00:00:00 +0000

Deployment Tools#

Publishing a project means moving assets off your laptop and into reliable environments. This module maps the tooling that packages content, automates delivery, and keeps services reachable for collaborators and users.

You’ll cover:

Content preparation by compressing images and authoring Markdown so sites load fast and stay readable.
Hosting options from static GitHub Pages sites and Google Colab notebooks to Vercel serverless functions, Docker and Podman containers, and GitHub Codespaces devcontainers.
Automation and connectivity using GitHub Actions CI/CD, secure ngrok tunnels, and REST APIs with correct CORS handling.
Production fundamentals including FastAPI backends, Google Auth integration, and local LLM endpoints with Ollama.

Together these tools let you launch prototypes quickly, iterate safely, and graduate data products into production-grade deployments.

Mon, 01 Jan 0001 00:00:00 +0000

Development Tools#

NOTE: The tools in this module are PRE-REQUISITES for the course. You would have used most of these before. If most of this is new to you, please take this course later.

Some tools are fundamental to data science because they are industry standards and widely used by data science professionals. Mastering these tools will align you with current best practices and making you more adaptable in a fast-evolving industry.

Mon, 01 Jan 0001 00:00:00 +0000

Browser: DevTools#

Chrome DevTools is the de facto standard for web development and data analysis in the browser. You’ll use this a lot when debugging and inspecting web pages.

Here are the key features you’ll use most:

Elements Panel
- Inspect and modify HTML/CSS in real-time
- Copy CSS selectors for web scraping
- Debug layout issues with the Box Model
```
// Copy selector in Console
copy($0); // Copies selector of selected element
```
Console Panel

Mon, 01 Jan 0001 00:00:00 +0000

Containers: Docker, Podman#

Docker and Podman are containerization tools that package your application and its dependencies into a standardized unit for software development and deployment.

Docker is the industry standard. Podman is compatible with Docker and has better security (and a slightly more open license). In this course, we recommend Podman but Docker works in the same way.

Initialize the container engine:

podman machine init
podman machine start

Common Operations. (You can use docker instead of podman in the same way.)

Mon, 01 Jan 0001 00:00:00 +0000

Embeddings: OpenAI and Local Models#

Embedding models convert text into a list of numbers. These are like a map of text in numerical form. Each number represents a feature, and similar texts will have numbers close to each other. So, if the numbers are similar, the text they represent mean something similar.

This is useful because text similarity is important in many common problems:

Search. Find similar documents to a query.
Classification. Classify text into categories.
Clustering. Group similar items into clusters.
Anomaly Detection. Find an unusual piece of text.

You can run embedding models locally or using an API. Local models are better for privacy and cost. APIs are better for scale and quality.

Mon, 01 Jan 0001 00:00:00 +0000

Extracting Audio and Transcripts#

Media Processing: FFmpeg#

FFmpeg is the standard command-line tool for processing video and audio files. It’s essential for data scientists working with media files for:

Extracting audio/video for machine learning
Converting formats for web deployment
Creating visualizations and presentations
Processing large media datasets

Basic Operations:

# Basic conversion
ffmpeg -i input.mp4 output.avi

# Extract audio
ffmpeg -i input.mp4 -vn output.mp3

# Convert format without re-encoding
ffmpeg -i input.mkv -c copy output.mp4

# High quality encoding (crf: 0-51, lower is better)
ffmpeg -i input.mp4 -preset slower -crf 18 output.mp4

Common Data Science Tasks:

Mon, 01 Jan 0001 00:00:00 +0000

Web Framework: FastAPI#

FastAPI is a modern Python web framework for building APIs with automatic interactive documentation. It’s fast, easy to use, and designed for building production-ready REST APIs.

Here’s a minimal FastAPI app, app.py:

# /// script
# requires-python = ">=3.11"
# dependencies = [
# "fastapi",
# "uvicorn",
# ]
# ///

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
async def root():
 return {"message": "Hello!"}


if __name__ == "__main__":
 import uvicorn

 uvicorn.run(app, host="0.0.0.0", port=8000)

Run this with uv run app.py.

Mon, 01 Jan 0001 00:00:00 +0000

May 2025 Feedback#

Based on feedback from 33 students, via ChatGPT .

Insights#

⭐ “Time-pressure” complainers actually rated ROE slightly higher. Students who mentioned ROE time limits gave ROE 2.61 vs 2.33 for others (+0.28). This suggests the most engaged cohort both feels the time squeeze and still perceives value—hinting the exam is closer to a “desirable difficulty” than a pure frustration. (Concept supported by learning-science literature on “desirable difficulties”.) (bjorklab.psych.ucla.edu )
Project-2 is love-it or hate-it. Median is 5/5 with many top scores, but variance is highest (σ≈1.54). Those flagging “difficulty/prereqs” rated P2 1.32 points lower than others—so P2 is excellent for prepared learners and punishing for under-prepared ones (bimodal fit).
Calling out TA/Discourse correlates with higher satisfaction. Mentions of TA/Discourse associate with slightly higher ratings across all four metrics (Δ overall +0.08, P1 +0.15, P2 +0.32, ROE +0.05). That’s counter to the intuition that “more TA talk = more complaints”; it reads as “the engaged students leaned on Discourse and benefited.”
Students know it’s hard—and still rate learning high. Despite many “hard/fast” comments, overall learning is strong (~4.0/5). That aligns with the course’s stated design (“ROE is hard”; “programming skills are a pre-requisite”), and suggests difficulty didn’t suppress perceived learning.
“Alignment” is a minority complaint—but a leverage point. Only a few explicitly said “align projects/ROE with taught content,” yet those who did had lower ROE ratings (−0.57) and overall learning (−0.47). A small group may be disproportionately dissatisfied due to scope drift.

Interesting observations#

ROE dominates the pain map. Time limit (45 min per the schedule), ambiguity, and “what exactly are we proving?” recur. A few ask for mock portals and clearer rubric.
P2 teaches beyond comfort. Many “pushes us beyond” / “real-world exposure” notes in “Best things,” while the “Worst” ask for more scaffolding. Classic split between stretch vs support.
Ops gaps show up: evaluation delays; release/schedule slips; a few network/portal reliability mentions; form UX complaints (Google Form structure).
Tooling friction: requests for dependable GPT/AI-Pipe access and deployment help; some struggled with “free versions” / rate limits. (AI-Pipe is in your ecosystem.)
Students explicitly ask for “instant feedback” on submissions (linting/auto-checks), not just grades—hinting appetite for continuous formative signals.
A minority asks for tougher projects (and longer ROE). The tails want even more challenge/time—a useful signal for differentiated tracks.
Desire for predictable cadence (earlier release, stick to schedule) appears in multiple corners even if counts are lower; these often co-occur with lower satisfaction.

Actions#

Gate early with a hard readiness check + branching track. Make GA-1 (already linked) the gate:
- If GA-1 < threshold: auto-route to a “Foundation track” of P2 (same outcomes, more scaffolds, smaller surface area).
- If GA-1 ≥ threshold: “Pro track” (current P2). Rationale: removes P2 bimodality; keeps challenge for prepared students; respects “desirable difficulty” while preventing wipeouts. Implementation is trivial (routing by score in your existing exam infra).
ROE redesign: keep pressure, fix friction.
- Keep 45-min “pressure-cooker” (it likely drives durable learning), but reduce non-constructive friction:
  - Two 10-min micro-mocks in the exact ROE environment the week prior (auto-graded, unlimited retries).
  - Stable prompt rubric with 2–3 canonical “what good looks like” exemplars.
  - Network-independent tasks (bundle data; pre-cache assets) and a simple “connectivity hiccup” auto-extend of 3–5 min if the portal detects time-outs. (bjorklab.psych.ucla.edu )
TA/Discourse “SLA + Triage Bot.”
- Publish an SLA (e.g., TA initial touch <6 h, resolution <24 h).
- Add a Discourse clarifier bot that: (i) de-dupes to canonical answers; (ii) proposes clearer wording for confusing prompts; (iii) tags to the right TA. (Start with a rules-only bot; graduate to LLM summaries later.) This leans into the positive correlation you’re already seeing.
Instant feedback everywhere.
- Add pre-submission checks: schema validators, smoke tests, minimal unit tests for projects; “red/green” badges on submit.
- For P2, expose a thin public checker (5–10 hidden tests retained for grading).
Cadence discipline via CI. Turn “release on time” from a promise into automation: a release calendar in the repo, GitHub Actions that publish content at fixed timestamps, and an automated grading queue dashboard (counts, average turnaround, SLA breaches).
Tooling reliability: AI-Pipe + deployment helpers.
- Provide a one-click “fallback” API key via AI-Pipe with rate-limit hints and a local stub; publish two minimal deployment recipes (Vercel/Cloudflare) with a pre-flight checklist. (Your repo already points to AI-Pipe.)
Micro-scaffolds for P2 only (don’t blunt the challenge). Ship optional scaffolds: sample input/output pairs, data dictionaries, 2 starter snippets. These trim unproductive thrash while preserving the tough core.
Rubric transparency + sample graded artifacts. Post 2 anonymized “A/B/C” exemplars with rubric mapping; students asked for “what exactly are we proving?” — this answers that.

Contrarian take:

Mon, 01 Jan 0001 00:00:00 +0000

Fine tuning LLMs: Gemini#

This material will likely be added in the May 2025 term. It is not part of the Jan 2025 term.

Mon, 01 Jan 0001 00:00:00 +0000

Forecasting with Excel#

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Function Calling with OpenAI#

Function Calling allows Large Language Models to convert natural language into structured function calls. This is perfect for building chatbots and AI assistants that need to interact with your backend systems.

OpenAI supports Function Calling – a way for LLMs to suggest what functions to call and how.

Here’s a minimal example using Python and OpenAI’s function calling that identifies the weather in a given location.

Mon, 01 Jan 0001 00:00:00 +0000

Geospatial Analysis with Excel#

You’ll learn how to create a data-driven story about coffee shop coverage in Manhattan, covering:

Data Collection: Collect and scrape data for coffee shop locations and census population from various sources.
Data Processing: Use Python libraries like geopandas for merging population data with geographic maps.
Map Creation: Generate coverage maps using tools like QGIS and Excel to visualize coffee shop distribution and population impact.
Visualization: Create physical, Power BI, and video visualizations to present the data effectively.
Storytelling: Craft a narrative around coffee shop competition, including strategic insights and potential market changes.

Here are links that explain how the video was made:

Mon, 01 Jan 0001 00:00:00 +0000

Geospatial Analysis with Python#

You’ll learn how to perform geospatial analysis for location-based decision making, covering:

Distance Calculation: Compute distances between various store locations and a reference point, such as the Empire State Building.
Data Visualization: Visualize store locations on a map using Python libraries like Folium.
Store Density Analysis: Determine the number of stores within a specified radius.
Proximity Analysis: Identify the closest and farthest stores from a specific location.
Decision Making: Use geospatial data to assess whether opening a new store is feasible based on existing store distribution.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Geospatial Analysis with QGIS#

You’ll learn how to use QGIS for geographic data processing, covering:

Shapefiles and KML Files: Create and manage shapefiles and KML files for storing and analyzing geographic information.
Downloading QGIS: Install QGIS on different operating systems and familiarize yourself with its interface.
Geospatial Data: Access and utilize shapefiles from sources like Diva-GIS and integrate them into QGIS projects.
Creating Custom Shapefiles: Learn how to create custom shapefiles when existing ones are unavailable, including creating a shapefile for South Sudan.
Editing and Visualization: Use QGIS tools to edit shapefiles, add attributes, and visualize geographic data with various styling and labeling options.
Exporting Data: Export shapefiles or KML files for use in other applications, such as Google Earth.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Version Control: Git, GitHub#

Git is the de facto standard for version control of software (and sometimes, data as well). It’s a system that keeps track of changes you make to files and folders. It allows you to revert to a previous state, compare changes, etc. It’s a central tool in any developer’s workflow.

GitHub is the most popular hosting service for Git repositories. It’s a website that shows your code, allows you to collaborate with others, and provides many useful tools for developers.

Mon, 01 Jan 0001 00:00:00 +0000

CI/CD: GitHub Actions#

GitHub Actions is a powerful automation platform built into GitHub. It helps automate your development workflow - running tests, deploying applications, updating datasets, retraining models, etc.

Understand the basics of YAML configuration files
Explore the pre-built actions from the marketplace
How to handle secrets securely
Triggering a workflow
Staying within the free tier limits
Caching dependencies to speed up workflows

Here is a sample .github/workflows/iss-location.yml that runs daily, appends the International Space Station location data into iss-location.jsonl, and commits it to the repository.

Mon, 01 Jan 0001 00:00:00 +0000

IDE: GitHub Codespaces#

GitHub Codespaces is a cloud-hosted development environment built right into GitHub that gets you coding faster with pre-configured containers, adjustable compute power, and seamless integration with workflows like Actions and Copilot.

Why Codespaces helps

Reproducible onboarding: Say goodbye to “works on my machine” woes—everyone uses the same setup for assignments or demos.
Anywhere access: Jump back into your project from a laptop, tablet, or phone without having to reinstall anything.
Rapid experimentation & debugging: Spin up short-lived environments on any branch, commit, or PR to isolate bugs or test features, or keep longer-lived codespaces for big projects.

Mon, 01 Jan 0001 00:00:00 +0000

AI Editor: GitHub Copilot#

AI Code Editors like GitHub Copilot , Cursor , Windsurf , Roo Code , Cline , Continue.dev , etc. use LLMs to help you write code faster.

Most are built on top of VS Code . These are now a standard tool in every developer’s toolkit.

GitHub Copilot is free (as of May 2025) for 2,000 completions and 50 chats.

You should learn about:

Mon, 01 Jan 0001 00:00:00 +0000

Static hosting: GitHub Pages#

GitHub Pages is a free hosting service that turns your GitHub repository directly into a static website whenever you push it. This is useful for sharing analysis results, data science portfolios, project documentation, and more.

Common Operations:

# Create a new GitHub repo
mkdir my-site
cd my-site
git init

# Add your static content
echo "<h1>My Site</h1>" > index.html

# Push to GitHub
git add .
git commit -m "feat(pages): initial commit"
git push origin main

# Enable GitHub Pages from the main branch on the repo settings page

Best Practices:

Mon, 01 Jan 0001 00:00:00 +0000

Google Authentication with FastAPI#

Secure your API endpoints using Google ID tokens to restrict access to specific email addresses.

Google Auth is the most commonly implemented single sign-on mechanism because:

It’s popular and user-friendly. Users can log in with their existing Google accounts.
It’s secure: Google supports OAuth2 and OpenID Connect to handle authentication.

Here’s how you build a FastAPI app that identifies the user.

Mon, 01 Jan 0001 00:00:00 +0000

Google Charts#

Google Chart 1 : How To Create Chart Or Graph On HTML CSS Website | Google Charts Tutorial
Google Chart 2 : Using Google Charts with googleVis

Mon, 01 Jan 0001 00:00:00 +0000

Google Data Studio#

Google Data Studio : Google Data Studio Tutorial for Beginners🔥

Mon, 01 Jan 0001 00:00:00 +0000

Hosting LLMs on Runpod#

This material will likely be added in the May 2025 term. It is not part of the Jan 2025 term.

Mon, 01 Jan 0001 00:00:00 +0000

HTTP Requests: curl, wget, HTTPie, Postman#

Making HTTP requests is essential for interacting with web APIs, downloading data, and testing web services. Whether you’re fetching data from a public API, testing your own REST endpoints, or automating data collection, these tools make HTTP interactions simple and efficient.

This guide covers four popular approaches: curl (the universal standard), wget (for downloads), HTTPie (user-friendly alternative), and Postman (GUI-based testing).

Watch these tutorials to understand HTTP requests and API testing (60 min):

Mon, 01 Jan 0001 00:00:00 +0000

Container hosting: Hugging Face Spaces with Docker#

Container platforms let you deploy applications in isolated, portable environments that include all dependencies. They’re perfect for ML applications that need custom environments, specific packages, or persistent state. Here are some common real-life uses:

A text summarization API that processes documents using custom NLP models (runs for 10-30 seconds per request)
An image classification tool that requires specific computer vision libraries (runs for 5-15 seconds per upload)
A chatbot with custom fine-tuned models that need GPU acceleration (runs for 2-5 seconds per message)

Unlike serverless functions, containers can maintain state, install any packages, write to the filesystem, and run background processes. They provide more control over the environment but require more configuration.

Mon, 01 Jan 0001 00:00:00 +0000

Hybrid Retrieval Augmented Generation (Hybrid RAG) with TypeSense#

Hybrid RAG combines semantic (vector) search with traditional keyword search to improve retrieval accuracy and relevance. By mixing exact text matches with embedding-based similarity, you get the best of both worlds: precision when keywords are present, and semantic recall when phrasing varies. TypeSense makes this easy with built-in hybrid search and automatic embedding generation.

Below is a fully self-contained Hybrid RAG tutorial using TypeSense, Python, and the command line.

Mon, 01 Jan 0001 00:00:00 +0000

Images: Compression#

Image compression is essential when deploying apps. Often, pages have dozens of images. Image analysis runs over thousands of images. The cost of storage and bandwidth can grow over time.

Here are things you should know when you’re compressing images:

Image dimensions are the width and height of the image in pixels. This impacts image size a lot
Lossless compression (PNG, WebP) preserves exact data
Lossy compression (JPEG, WebP) removes some data for smaller files
Vector formats (SVG) scale without quality loss
WebP is the modern standard, supporting both lossy and lossless

Here’s a rule of thumb you can use as of 2025.

Mon, 01 Jan 0001 00:00:00 +0000

JSON#

JSON (JavaScript Object Notation) is the de facto standard format for data exchange on the web and APIs. Its human-readable format and widespread support make it essential for data scientists working with web services, APIs, and configuration files.

For data scientists, JSON is essential when:

Working with REST APIs and web services
Storing configuration files and metadata
Parsing semi-structured data from databases like MongoDB
Creating data visualization specifications (e.g., Vega-Lite)

Watch this comprehensive introduction to JSON (15 min):

Mon, 01 Jan 0001 00:00:00 +0000

Large Language Models#

This module covers the practical usage of large language models (LLMs).

LLMs incur a cost. For the May 2025 batch, use aipipe.org as a proxy. Emails with @ds.study.iitm.ac.in get a $1 per calendar month allowance. (Don’t exceed that.)

Read the AI Pipe documentation to learn how to use it. But in short:

Replace OPENAI_BASE_URL, i.e. https://api.openai.com/v1 with https://aipipe.org/openrouter/v1... or https://aipipe.org/openai/v1...
Replace OPENAI_API_KEY with the AIPIPE_TOKEN
Replace model names, e.g. gpt-4.1-nano, with openai/gpt-4.1-nano

For example, let’s use Gemini 2.0 Flash Lite via OpenRouter for chat completions and Text Embedding 3 Small via OpenAI for embeddings:

Mon, 01 Jan 0001 00:00:00 +0000

Live Sessions#

Live sessions by the instructors and TAs are recorded and uploaded to the YouTube Tools in Data Science channel .

These are transcribed by faq.sh into FAQ-style markdown files, which are stored here.

Sessions#

Video | FAQ | 2025 10 04 Project 1 - Session 1 TDS Sep 2025 (2h 24m)
Video | FAQ | 2025 10 03 week 2 - Session 4 TDS Sep 2025 (2h 16m)
Video | FAQ | 2025 09 30 week 2 - Session 3 TDS Sep 2025 (1h 45m)
Video | FAQ | 2025 09 28 week 2 - Session 2 TDS Sep 2025 (2h 36m)
Video | FAQ | 2025 09 27 week 2 - Session 1 TDS Sep 2025 (2h 53m)
Video | FAQ | 2025 09 24 week 1 - Session 5 TDS Sep 2025 (1h 41m)
Video | FAQ | 2025 09 29 Week 1 - Session 4 TDS Sep 2025 (1h 38m)
Video | FAQ | 2025 09 21 Week 1 - Session 2 TDS Sep 2025 (2h 37m)
Video | FAQ | 2025 09 22 Week 1 - Session 3 TDS Sep 2025 (1h 58m)
Video | FAQ | 2025 09 20 Week 1 - Session 1 TDS Sep 2025 (2h 52m)
Video | FAQ | 2025 08 04 project 2 - Q&A 2 TDS May 2025 (1h 59m)
Video | FAQ | 2025 07 26 project 2 - Session 2 TDS May 2025 (1h 42m)
Video | FAQ | 2025 07 24 project 2 - Session 1 TDS May 2025 (1h 50m)
Video | FAQ | 2025 07 21 project 2 - Q&A 1 TDS May 2025 (1h 32m)
Video | FAQ | 2025 07 18 MOCK ROE session 4 - TDS May 2025 (2h 7m)
Video | FAQ | 2025 07 17 MOCK ROE session 3 - TDS May 2025 (1h 33m)
Video | FAQ | 2025 07 16 MOCK ROE session 2 - TDS May 2025 (0h 53m)
Video | FAQ | 2025 07 10 MOCK ROE session1 - TDS May 2025 (1h 56m)
Video | FAQ | 2025 07 06 Week 6 Session 2 - TDS May 2025 (1h 49m)
Video | FAQ | 2025 07 05 Week 6 Session 1 - TDS May 2025 (1h 17m)
Video | FAQ | 2025 06 28 Week 5 Session 1 - TDS May 2025 (2h 8m)
Video | FAQ | 2025 06 15 Week 4 1 Session 1 - TDS May 2025 (1h 26m)
Video | FAQ | 2025 06 12 Project 1 - DCS TDS May 2025 (1h 2m)
Video | FAQ | 2025 06 12 Project 1 Session 3 - TDS May 2025 (2h 26m)
Video | FAQ | 2025 06 11 Project 1 Session 2 - TDS May 2025 (2h 59m)
Video | FAQ | 2025 06 10 Project 1 Session 1 - TDS May 2025 (2h 28m)
Video | FAQ | 2025 05 29 Week 3 Session 4 - TDS May 2025 (2h 44m)
Video | FAQ | 2025 05 28 Week 3 Session 3 - TDS May 2025 (1h 30m)
Video | FAQ | 2025 05 25 Week 3 Session 2 - TDS May 2025 (1h 38m)
Video | FAQ | 2025 05 24 Week 3 Session 1 - TDS May 2025 (1h 47m)
Video | FAQ | 2025 05 23 Week 2 Session 2 - Github Actions, Docker - TDS May 2025 (3h 17m)
Video | FAQ | 2025 05 22 Week 2 Session 2 - Local Lamma, Ngrok, Google Authentication TDS May 2025 (2h 40m)
Video | FAQ | 2025 05 21 Week 2 Session 1 TDS May 2025 (2h 31m)
Video | FAQ | 2025 05 17 Week 1 Session 5 TDS May 2025 (2h 37m)
Video | FAQ | 2025 05 16 Week 1 Session 4 Env Settings, Env Variables, git and github TDS May 2025 (2h 12m)
Video | FAQ | 2025 05 15 Week 1 Session 3 Linux, BASH, git & github TDS May 2025 (3h 3m)
Video | FAQ | 2025 05 11 Week 1 Session 2 Setup & Debugging TDS May 2025 (3h 3m)
Video | FAQ | 2025-05-10 Week 1 - Session 1: Introduction to TDS - TDS May 25 (2h 29m)
Video | FAQ | 2025-04-09 Live Session - Qs & As - TDS Jan 25 (1h 19m)
Video | FAQ | 2025-04-08 End Term Session 1 - TDS Jan 25 (1h 59m)
Video | FAQ | 2025-03-27 Week 11 - Session 3 - TDS Jan 25 (2h 7m)
Video | FAQ | 2025-03-26 Week 11 - Session 2 - TDS Jan 25 (1h 27m)
Video | FAQ | 2025-03-25 Week 11 - Session 1 - TDS Jan 25 (1h 17m)
Video | FAQ | 2025-03-20 Week 10 - Session 3 - TDS Jan 25 (1h 44m)
Video | FAQ | 2025-03-18 Week 10 - Session 1 - TDS Jan 25 (2h 24m)
Video | FAQ | 2025-03-13 Week 9 - Session 3 - TDS Jan 25 (1h 58m)
Video | FAQ | 2025-03-11 Week 9 - Session 1 - TDS Jan 25 (2h 11m)
Video | FAQ | 2025-03-12 Week 9 - Session 2 - TDS Jan 25 (1h 14m)
Video | FAQ | 2025-03-04 Week 8 - Session 1 - TDS Jan 25 (1h 52m)
Video | FAQ | 2025-03-04 Week 8 - Session 3 - TDS Jan 25 (1h 52m)
Video | FAQ | 2025-03-04 Week 8 - Session 2 - TDS Jan 25 (0h 47m)
Video | FAQ | 2025-02-27 Week 7 - Session 3 - TDS Jan 25 (2h 24m)
Video | FAQ | 2025-02-26 Week 7 - Session 2 - TDS Jan 25 (5h 8m)
Video | FAQ | TA’s Session TDS – 2024 11 10 19 49 (1h 39m)
Video | FAQ | 2025-02-18 Week 6 - Session 3 - TDS Jan 25 (1h 34m)
Video | FAQ | 2025-02-18 Week 6 - Session 1 - TDS Jan 25 (2h 8m)
Video | FAQ | 2025-02-13 Week 5 - Session 3 - TDS Jan 25 (3h 12m)
Video | FAQ | 2025-02-11 Week 5 - Session 1 - TDS Jan 25 (3h 57m)
Video | FAQ | 2025-02-06 Week 4 - Session 4 - TDS Jan 25 (2h 16m)
Video | FAQ | 2025-02-06 Week 4 - Session 3 - TDS Jan 25 (2h 24m)
Video | FAQ | 2025-02-04 Week 4 - Session 1 - TDS Jan 25 (1h 27m)
Video | FAQ | 2025-02-01 Week 3 - Session 5 - TDS Jan 25 (2h 34m)
Video | FAQ | 2025-01-31 Week 3 - Session 4 - TDS Jan 25 (2h 45m)
Video | FAQ | 2025-01-30 Week 3 - Session 3 - TDS Jan 25 (2h 35m)
Video | FAQ | 2025-01-29 Week 3 - Session 2 - TDS Jan 25 (1h 28m)
Video | FAQ | 2025-01-28 Week 3 - Session 1 - TDS Jan 25 (2h 0m)
Video | FAQ | 2025-01-23 Week 2 - Session 4 - TDS Jan 25 (2h 5m)
Video | FAQ | 2025-01-22 Week 2 - Session 3 - TDS Jan 25 (1h 19m)
Video | FAQ | 2025-01-21 Week 2 - Session 2 - TDS Jan 25 (2h 15m)
Video | FAQ | 2025-01-20 Week 2 - Session 1 - TDS Jan 25 (2h 3m)
Video | FAQ | 2025-01-17 Week 1 - Session 2 - TDS Jan 25 (2h 17m)
Video | FAQ | 2025-01-16 Week 1 - Session 1 - TDS Jan 25 (1h 59m)
Video | FAQ | 2025-01-15 Introduction to Tools for Data Science Jan 25 (1h 18m)
Video | FAQ | Mock QP - TA Session - TDS 2024 12 19 18 51 (2h 25m)
Video | FAQ | Project 2 TA’s Session TDS 2024 12 05 19 52 (3h 0m)
Video | FAQ | TA’s Session TDS 2024 12 05 19 52 GMT+05 30 Recording (3h 0m)
Video | FAQ | ROE Session TDS 2024 11 16 19 52 (1h 13m)
Video | FAQ | TA’s Session TDS – 2024 11 07 19 58 GMT+05 30 – How to hack Project 1 and possibly ROE, pdf scrapi (2h 14m)
Video | FAQ | ROE Session TDS 2024 11 15 19 53 (2h 0m)
Video | FAQ | TA’s Session TDS – 2024 11 10 19 49 GMT+05 30 – Recording (1h 39m)
Video | FAQ | TA’s Session TDS 2024 11 14 19 54 (1h 52m)
Video | FAQ | TA’s Session TDS – 2024 11 04 16 56 IST – Recording (1h 55m)
Video | FAQ | TA’s Session TDS – 2024 10 27 19 55 IST – Recording (1h 45m)
Video | FAQ | TA’s Session TDS – 2024 10 24 19 50 IST – Recording 2 (2h 41m)
Video | FAQ | TA’s Session TDS – 2024-10-20 19:50 IST – Recording (1h 59m)
Video | FAQ | TAs Session - 2024 10 17 20:09:44 (1h 50m)
Video | FAQ | TA’s Session TDS – 2024 10 13 19 49 IST – Recording (2h 11m)
Video | FAQ | TA’s Session TDS – 2024 10 06 19 54 IST – Recording (1h 35m)
Video | FAQ | TA’s Session TDS – 2024 09 29 19 49 IST – Recording (1h 40m)
Video | FAQ | TA Session 29th August: Revision: Modules 7 to 9, CSS selectors, Javascript (1h 25m)
Video | FAQ | TA Session 25th August: Revision: Modules 4 to 6 (1h 51m)
Video | FAQ | TA Session 22 August: Mock and End Term, Revision: Modules 1 to 3 (2h 6m)
Video | FAQ | TA Session 19 August: Streamlit, Heroku (1h 47m)
Video | FAQ | TA Session 19 July: ROE Survival : Part 2 (2h 19m)
Video | FAQ | TA Session 18 July: The best ROE Survival Strategies, Tricks and Tips: Part 1 (2h 12m)
Video | FAQ | “TA session 12 July : week 5 , LLM API and embedding” (1h 54m)
Video | FAQ | TA session 14 July : ROE intro and strategy (1h 15m)
Video | FAQ | TA Session 11 July: LLM APi Setup and Use (2h 48m)
Video | FAQ | TA Session 7 July: Project Discussion, and Excel Filters, Slope Regression (1h 51m)
Video | FAQ | TA Session 4 July: More Pandas, Parquet processing, Forecast.ETS, Database Processing and Week 4 GA (1h 48m)
Video | FAQ | TA Session 30 June : More complex webscraping (2h 5m)
Video | FAQ | TA Session 27 June : Processing files with Pandas, Open Refine and Week 3 Graded Assignment (1h 51m)
Video | FAQ | TA session 20 June : Week 2 concepts ,API, JSON, PDF scraping and GA discussion (2h 32m)
Video | FAQ | A session 23 June : Web scraping with cookies and understanding JSON (1h 57m)
Video | FAQ | TA Session 16 June : Introduction to HTML and web scraping (2h 7m)
Video | FAQ | TA session 13 June : Introduction and week 1,2 recap (2h 2m)
Video | FAQ | ET revision : Attribute related to Anonymization and autocorrelation plot (1h 2m)
Video | FAQ | TA session : PYQ April 23 discussion part - 2 (2h 9m)
Video | FAQ | TA session : PYQ April 23 discussion part - 1 (1h 44m)
Video | FAQ | TA session : PYQ Dec 23 discussion (1h 12m)
Video | FAQ | 05 ROE revision session jan 24 (2h 56m)
Video | FAQ | 04 - XML intro and scraping (2h 3m)
Video | FAQ | 02 - Fundamentals of web scraping with urllib and BeautifulSoup (1h 37m)
Video | FAQ | 03 - Intermediate web scraping use of cookies (2h 19m)
Video | FAQ | 01 - Intro to Web scraping and HTML (1h 33m)
Video | FAQ | Live Session - 2 (0h 29m)

Mon, 01 Jan 0001 00:00:00 +0000

Local LLMs: Llamafile#

You would have heard of Large Language Models (LLMs) like GPT-4, Claude, and Llama. Some of these models are available for free, but most of them are not.

An easy way to run LLMs locally is Mozilla’s Llamafile . It’s a single executable file that works on Windows, Mac, and Linux. No installation or configuration needed - just download and run.

Watch this Llamafile Tutorial (6 min):

Mon, 01 Jan 0001 00:00:00 +0000

LLM Agents: Building AI Systems That Can Think and Act#

LLM Agents are AI systems that can define and execute their own workflows to accomplish tasks. Unlike simple prompt-response patterns, agents make multiple LLM calls, use tools, and adapt their approach based on intermediate results. They represent a significant step toward more autonomous AI systems.

What Makes an Agent?#

An LLM agent consists of three core components:

Mon, 01 Jan 0001 00:00:00 +0000

LLM Evaluations with PromptFoo#

Test-drive your prompts and models with automated, reliable evaluations.

PromptFoo is a test-driven development framework for LLMs:

Developer-first: Fast CLI with live reload & caching (promptfoo.dev )
Multi-provider: Works with OpenAI, Anthropic, HuggingFace, Ollama & more (GitHub )
Assertions: Built‑in (contains, equals) & model‑graded (llm-rubric) (docs )
CI/CD: Integrate evals into pipelines for regression safety (CI/CD guide )

To run PromptFoo:

Mon, 01 Jan 0001 00:00:00 +0000

Gemini Flash Experimental Image Generation and Editing APIs#

In March 2025, Google introduced native image generation and editing capabilities in the Gemini 2.0 Flash Experimental model. You can now generate and iteratively edit images via a single REST endpoint (Experiment with Gemini 2.0 Flash native image generation , Generate images | Gemini API | Google AI for Developers ).

(How to use Latest Gemini 2.0 Native Image Generation with API? )

Mon, 01 Jan 0001 00:00:00 +0000

LLM Sentiment Analysis#

OpenAI’s API provides access to language models like GPT 4o, GPT 4o mini, etc.

For more details, read OpenAI’s guide for:

Start with this quick tutorial:

Here’s a minimal example using curl to generate text:

curl https://api.openai.com/v1/chat/completions \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 -d '{
 "model": "gpt-4o-mini",
 "messages": [{ "role": "user", "content": "Write a haiku about programming." }]
 }'

Let’s break down the request:

Mon, 01 Jan 0001 00:00:00 +0000

OpenAI TTS-1 for Text-to-Speech Generation#

OpenAI’s Text-to-Speech API (TTS-1) converts text into natural-sounding speech using state-of-the-art neural models. Released in March 2025, it offers multiple voices and control over speaking style and speed.

Simple speech generation#

To generate speech from text, send a POST request to the speech endpoint:

curl https://api.openai.com/v1/audio/speech \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 -H "Content-Type: application/json" \
 -d '{
 "model": "tts-1",
 "input": "Hello! This is a test of the OpenAI text to speech API.",
 "voice": "alloy"
 }' --output speech.mp3

Generation options#

Control the output with these parameters:

Mon, 01 Jan 0001 00:00:00 +0000

LLM Text Extraction#

JSON is one of the most widely used formats in the world for applications to exchange data.

This video explains how to use LLMs to extract structure from unstructured data, covering:

LLM for Data Extraction: Use OpenAI’s API to extract structured information from unstructured data like addresses.
JSON Schema: Define a JSON schema to ensure consistent and structured output from the LLM.
Prompt Engineering: Craft effective prompts to guide the LLM’s response and improve accuracy.
Data Cleaning: Use string functions and OpenAI’s API to clean and standardize data.
Data Analysis: Analyze extracted data using Pandas to gain insights.
LLM Limitations: Understand the limitations of LLMs, including potential errors and inconsistencies in output.
Production Use Cases: Explore real-world applications of LLMs for data extraction, such as customer service email analysis.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

LLM Video Screen-Scraping#

Video screen-scraping with LLMs is a powerful technique for extracting structured data from screen recordings. This approach works with any visible screen content and bypasses traditional web scraping limitations like authentication or anti-scraping measures.

Key benefits:

No setup cost or authentication handling
Works with any visible screen content
Full control over data exposure
Extremely cost-effective (< $0.001 per short video)
Bypasses anti-scraping measures
Handles varying formats and layouts

Quick Start Example#

Here’s a basic workflow using Google’s AI Studio and Gemini:

Mon, 01 Jan 0001 00:00:00 +0000

LLM Website Scraping#

LLM-powered website scraping represents a paradigm shift from traditional web scraping. Instead of writing explicit code to navigate DOM structures and CSS selectors, you describe what data you want and let the LLM figure out how to extract it.

This approach works for both simple public pages and complex scenarios requiring authentication, JavaScript rendering, and dynamic interactions. LLMs can write scraping code, control browsers, adapt to layout changes, and even handle captchas through browser automation.

Mon, 01 Jan 0001 00:00:00 +0000

LLM CLI: llm#

llm is a command-line utility for interacting with large language models—simplifying prompts, managing models and plugins, logging every conversation, and extracting structured data for pipelines.

Basic Usage#

Install llm . Then set up your OPENAI_API_KEY environment variable. See Getting started .

TDS Students: See Large Language Models for instructions on how to get and use OPENAI_API_KEY.

# Run a simple prompt
llm 'five great names for a pet pelican'

# Continue a conversation
llm -c 'now do walruses'

# Start a memory-aware chat session
llm chat

# Specify a model
llm -m gpt-4.1-nano 'Summarize tomorrow’s meeting agenda'

# Extract JSON output
llm 'List the top 5 Python viz libraries with descriptions' \
 --schema-multi 'name,description'

Or use llm without installation using uvx :

Mon, 01 Jan 0001 00:00:00 +0000

Making Open Data Useful: Lessons from Diagram Chasing#

Many datasets are technically available but practically unusable. Your job is to make them useful for people who are not data scientists. Design for the lowest friction, and let people share one precise slice of truth with a link.

See how Diagram Chasing does this across projects like CBFC Watch, Time Use Explorer, BLR Water Log, Votes in a name?, and Who is my neta? (CBFC Watch )

Mon, 01 Jan 0001 00:00:00 +0000

Interactive Notebooks: Marimo#

Marimo is a new take on notebooks that solves some headaches of Jupyter. It runs cells reactively - when you change one cell, all dependent cells update automatically, just like a spreadsheet.

Marimo’s cells can’t be run out of order. This makes Marimo more reproducible and easier to debug, but requires a mental shift from the Jupyter/Colab way of working.

It also runs Python directly in the browser and is quite interactive. Browse the gallery of examples . With a wide variety of interactive widgets, It’s growing popular as an alternative to Streamlit for building data science web apps.

Mon, 01 Jan 0001 00:00:00 +0000

Documentation: Markdown#

Markdown is a lightweight markup language for creating formatted text using a plain-text editor. It’s the standard for documentation in software projects and data science notebooks.

Watch this introduction to Markdown (19 min):

Common Markdown syntax:

# Heading 1
## Heading 2

**bold** and *italic*

- Bullet point
- Another point
 - Nested point

1. Numbered list
2. Second item

[Link text](https://url.com)
![Image alt](image.jpg)

```python
# Code block
def hello():
 print("Hello")
```

> Blockquote

There is also a GitHub Flavored Markdown standard which is popular. This includes extensions like:

Mon, 01 Jan 0001 00:00:00 +0000

Marks Dashboard#

T - Score Calculation#

The final score for the course is calculated as follows:

Components (All out of 100)	Weight
Grade Assessments (GA) Best 5/8	20%
Project 1 (P1)	20%
Project 2 (P2)	20%
Remote Online Exam (ROE)	20%
End Term (ET)	20%

End Term Eligibility: Best 4/5 first GAs Average >= 40#

T = 0.2 GA + 0.2 P1 + 0.2 P2 + 0.2 ROE + 0.2 ET

Mon, 01 Jan 0001 00:00:00 +0000

Marp: Markdown Presentation Ecosystem#

Marp (Markdown Presentation) is a powerful tool for creating presentations using Markdown. It converts Markdown files into slideshows, making it ideal for:

Technical Presentations: Code snippets, diagrams, and technical content
Documentation: Creating slide decks from existing documentation
Academic Slides: Research presentations with math equations and citations
Conference Talks: Professional presentations with custom themes
Teaching Materials: Educational content with rich formatting
GitHub Pages: Hosting presentations on the web

This tutorial covers using Marp in VS Code to create professional presentations.

Mon, 01 Jan 0001 00:00:00 +0000

Multimodal Embeddings#

Multimodal embeddings map text and images into the same vector space, enabling direct comparison between, say, a caption — “A cute cat” — and an image of that cat. This unified representation powers real-world applications like:

Cross-modal search (e.g. “find images of a sunset” via text queries)
Content recommendation (suggesting visually similar products to text descriptions)
Clustering & retrieval (grouping documents and their associated graphics)
Anomaly detection (spotting unusual image–text pairings)

By reducing different data types to a common numeric form, you unlock richer search, enhanced recommendations, and tighter integration of visual and textual data.

Mon, 01 Jan 0001 00:00:00 +0000

Narratives with Comics#

Sample sheet

Mon, 01 Jan 0001 00:00:00 +0000

Narratives with Excel#

Sample Excel

Mon, 01 Jan 0001 00:00:00 +0000

Network Analysis in Python#

You’ll learn how to use network analysis to identify clusters and connections between nodes in a dataset, covering:

Network Construction: Build a network from the IMDb database, where nodes represent actors and edges represent shared movie appearances.
Clustering: Apply clustering techniques to detect communities within the network, using the scikit-network library.
Matrix Operations: Utilize matrix operations to efficiently analyze actor relationships and interactions.
Community Detection: Implement algorithms to identify and interpret clusters, examining how different actor clusters are connected.
Application of Findings: Explore practical applications of network analysis, such as social network analysis and its potential uses in various domains.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Tunneling: ngrok#

Ngrok is a tool that creates secure tunnels to your localhost, making your local development server accessible to the internet. It’s essential for testing webhooks, sharing work in progress, or debugging applications in production-like environments.

Run the command uvx ngrok http 8000 to create a tunnel to your local server on port 8000. This generates a public URL that you can share with others.

Mon, 01 Jan 0001 00:00:00 +0000

Nominatim API with Python#

You’ll learn how to get the latitude and longitude of any city from the Nominatim API.

Introduction to Nominatim: Understand how Nominatim, from OpenStreetMap, works similarly to Google Maps for geocoding.
Installation and Import: Learn to install and import geopy and nominatim .
Using the Locator: Create a locator object using Nominatim and set up a user agent.
Geocoding an Address: Use locator.geocode to input an address (e.g., Eiffel Tower) and fetch geocoded data.
Extracting Data: Access detailed information like latitude, longitude, bounding box, and accurate address from the JSON response.
Classifying Locations: Identify the type of place (e.g., tourism, university) using the response data.
Practical Example: Geocode “IIT Madras” and retrieve its full address, type (university), and other relevant information.

Here are links and references:

Mon, 01 Jan 0001 00:00:00 +0000

JavaScript tools: npx#

npx is a command-line tool that comes with npm (Node Package Manager) and allows you to execute npm package binaries and run one-off commands without installing them globally. It’s essential for modern JavaScript development and data science workflows.

For data scientists, npx is useful when:

Running JavaScript-based data visualization tools
Converting notebooks and documents
Testing and formatting code
Running development servers

Here are common npx commands:

# Run a package without installing
npx http-server . # Start a local web server
npx prettier --write . # Format code or docs
npx eslint . # Lint JavaScript
npx ts-node script.ts # Run TypeScript directly
npx esbuild app.js # Bundle JavaScript
npx jsdoc . # Generate JavaScript docs

# Run specific versions
npx prettier@3.6 --write . # Use prettier 3.6

# Execute remote scripts (use with caution!)
npx github:user/repo # Run from GitHub

Watch this introduction to npx (6 min):

Mon, 01 Jan 0001 00:00:00 +0000

Local LLM Runner: Ollama#

ollama is a command-line tool for running open-source large language models entirely on your own machine—no API keys, no vendor lock-in, full control over models and performance.

Basic Usage#

Download Ollama for macOS, Linux, or Windows and add the binary to your PATH. See the full Docs ↗ for installation details and troubleshooting.

# List installed and available models
ollama list

# Download/pin a specific model version
ollama pull gemma3:1b-it-qat

# Run a one-off prompt
ollama run gemma3:1b-it-qat 'Write a haiku about data visualization'

# Launch a persistent HTTP API on port 11434
ollama serve

# Interact programmatically over HTTP
curl -X POST http://localhost:11434/api/chat \
 -H 'Content-Type: application/json' \
 -d '{"model":"gemma3:1b-it-qat","prompt":"Hello, world!"}'

Key Features#

Model management: list/pull — Install and switch among Llama 3.3, DeepSeek-R1, Gemma 3, Mistral, Phi-4, and more.
Local inference: run — Execute prompts entirely on-device for privacy and zero latency beyond hardware limits.
Persistent server: serve — Expose a local REST API for multi-session chats and integration into scripts or apps.
Version pinning: pull model:tag — Pin exact model versions for reproducible demos and experiments.
Resource control: --threads / --context — Tune CPU/GPU usage and maximum context window for performance and memory management.

Real-World Use Cases#

Quick prototyping. Brainstorm slide decks or blog outlines offline, without worrying about API quotas: ollama run gemma-3 'Outline a slide deck on Agile best practices'
Data privacy. Summarize sensitive documents on-device, retaining full control of your data: cat financial_report.pdf | ollama run phi-4 'Summarize the key findings'
CI/CD integration. Validate PR descriptions or test YAML configurations in your pipeline without incurring API costs: git diff origin/main | ollama run llama2 'Check for style and clarity issues'
Local app embedding. Power a desktop or web app via the local REST API for instant LLM features: curl -X POST http://localhost:11434/api/chat -d '{"model":"mistral","prompt":"Translate to German"}'

Read the full Ollama docs ↗ for advanced topics like custom model hosting, GPU tuning, and integrating with your development workflows.

Mon, 01 Jan 0001 00:00:00 +0000

Outlier Detection with Excel#

You’ll learn how to identify and handle outliers in data using Excel, covering:

Understanding Outliers: Definition of outliers and their impact on statistical analysis.
Calculating Quartiles: Using Excel formulas to calculate Q1 (first quartile) and Q3 (third quartile).
Interquartile Range (IQR): Finding the IQR by subtracting Q1 from Q3.
Determining Bounds: Calculating lower and upper bounds using 1.5 times the IQR.
Identifying Outliers: Using Excel functions to determine if data points fall outside the calculated bounds.
Visualizing Data: Creating box plots to visualize outliers and data distribution.
Handling Outliers: Deciding whether to exclude or keep outliers based on their impact on analysis.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Parsing JSON#

JSON is everywhere—APIs, logs, configuration files—and its nested or large structure can challenge memory and processing. In this tutorial, we’ll explore tools to flatten, stream, and query JSON data efficiently.

For example, we’ll often need to process a multi-gigabyte log file from a web service where each record is a JSON object.

This requires us to handle complex nested structures, large files that don’t fit in memory, or extract specific fields. Here are the key tools and techniques for efficient JSON parsing:

Mon, 01 Jan 0001 00:00:00 +0000

Profile Data with Python#

This session covers the use of the pandas_profiling library for generating comprehensive data reports in Python:

Library Installation and Import: Learn how to install and import the pandas_profiling library.
Profile Report Generation: Generate an HTML report with a single line of code using ProfileReport.
Descriptive Statistics: View detailed descriptive statistics such as variance, standard deviation, and kurtosis.
Outlier Detection: Identify and analyze outliers within the dataset.
Correlation Analysis: Understand how variables are correlated with each other using visual representations.
Handling Missing Values: Get insights on missing data and decide on imputation or removal strategies.
Initial Data Insights: Use the report to gather early warnings and insights before starting the data cleaning and modeling process.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Project: Data Analyst Agent#

Deploy a data analyst agent. This is an API that uses LLMs to source, prepare, analyze, and visualize any data.

Your application exposes an API endpoint. You may host it anywhere. Let’s assume it’s at https://app.example.com/api/.

The endpoint must accept a POST request, e.g. POST https://app.example.com/api/ with a data analysis task description and optional attachments in the body. For example:

curl "https://app.example.com/api/" -F "questions.txt=@question.txt" -F "image.png=@image.png" -F "data.csv=@data.csv"

questions.txt will ALWAYS be sent and contain the questions. There may be zero or more additional files passed.

Mon, 01 Jan 0001 00:00:00 +0000

Project: LLM Code Deployment#

In this project, students will build an application that can build, deploy, update an application!

Build. The student:
- receives & verifies a request containing an app brief
- uses an LLM-assisted generator to build the app,
- deployes to GitHub Pages,
- then pings an evaluation API with repo details
Evaluate. The instructors:
- run automated static, dynamic (Playwright), and and LLM checks
- store and publish the results after the deadline
- send a second request tailored to the student’s codebase
Revise. The student
- verifies secret
- updates the app based on the request
- re‑deploys Pages
- then pings a second evaluation API with repo metadata.

Request#

The request is a JSON file like this:

Mon, 01 Jan 0001 00:00:00 +0000

Project: TDS Virtual TA#

Create a virtual Teaching Assistant Discourse responder.

Background#

You are a clever student who has joined IIT Madras’ Online Degree in Data Science. You have just enrolled in the Tools in Data Science course.

Out of kindness for your teaching assistants, you have decided to build an API that can automatically answer student questions on their behalf based on:

Course content with content for TDS Jan 2025 as on 15 Apr 2025.
TDS Discourse posts with content from 1 Jan 2025 - 14 Apr 2025.

Scrape the data#

To make sure you can answer these questions, you will need to extract the data from the above source.

Mon, 01 Jan 0001 00:00:00 +0000

Prompt Engineering#

Prompt engineering is the process of crafting effective prompts for large language models (LLMs).

One of the best ways to approach prompt engineering is to think of LLMs as a smart colleague (with amnesia) who needs explicit instructions.

The most authoritative guides are from the LLM providers themselves:

Here are some best practices:

Use prompt optimizers#

They rewrite your prompt to improve it. Explore:

Be clear, direct, and detailed#

Be explicit and thorough. Include all necessary context, goals, and details so the model understands the full picture.

Mon, 01 Jan 0001 00:00:00 +0000

Pydantic AI#

Pydantic is Python’s most widely used data validation library. Pydantic AI extends this to build production-grade AI agents with structured outputs and type safety.

What’s Pydantic?#

Python doesn’t enforce type hints at runtime. Pydantic validates data automatically using type annotations:

# /// script
# requires-python = ">=3.11"
# dependencies = ["pydantic"]
# ///

from pydantic import BaseModel, Field


class User(BaseModel):
 name: str
 age: int = Field(..., gt=0, le=120)
 email: str


# Automatic validation and type conversion
user = User(name="Alice", age="25", email="alice@example.com")
print(user.age, type(user.age)) # 25 <class 'int'>

# Invalid data raises ValidationError
try:
 User(name="Bob", age=-5, email="invalid")
except Exception as e:
 print(f"Validation failed: {e}")

Pydantic is used by FastAPI, LangChain, the OpenAI SDK, and 8,000+ packages on PyPI. It’s fast (core written in Rust), integrates with IDEs, and generates JSON schemas automatically.

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval Augmented Generation (RAG) with the CLI#

Retrieval Augmented Generation (RAG) combines retrieval (searching a knowledge base) with generation (using an LLM) to produce answers grounded in your own documents. Instead of relying solely on a general-purpose LLM, RAG lets you feed it the most relevant chunks from your corpus at query time, improving accuracy, reducing hallucinations, and allowing you to answer domain‑specific questions without fine‑tuning.

In particular, you can answer questions that are hard to answer with a keyword search. For example:

Mon, 01 Jan 0001 00:00:00 +0000

RAWgraphs#

RAWgraphs

You’ll learn about RAWGraphs and how to create data visualizations, covering:

Understanding RAWGraphs: Learn that RAWGraphs is an open-source data visualization tool designed to simplify the visualization of complex data for everyone.
Data Import: Discover how to import your dataset into RAWGraphs by copying and pasting it from any spreadsheet application.
Chart Selection: Understand how to choose from a variety of charts that are built on top of d3js, such as a scatter plot, suitable for your data.
Chart Creation: Create different kinds of charts, notably non-traditional charts that go beyond bars, lines, pie, etc.
Mapping Data Dimensions: Learn the process of mapping your dataset’s dimensions to the visual variables of your chosen chart. This includes examples like mapping rating to the horizontal axis, production budget to the vertical axis, box office to bubble areas, genre to color, and movie names to labels.
Customization and Export: Find out how to customize your visualization and export it as a vector or raster image, which can then be fine-tuned using any graphic editor.
Further Resources: Be informed about where to find more information on RAWGraphs’ website and how to contribute to the project through GitHub.

Watch these videos to learn how to create non-traditional data visualizations with RawGraphs .

Mon, 01 Jan 0001 00:00:00 +0000

Regression with Excel#

You’ll learn to perform regression analysis using Excel, covering:

Data Preparation: Understanding the cleaned dataset and necessary columns for analysis.
Enabling the Tool: How to enable the Data Analysis Tool Pack in Excel.
Types of Regression: Differences between simple and multiple linear regression.
Setting Up Regression: Steps to input dependent (new deaths) and independent variables (new cases, new tests, new vaccinations, stringency index) for the analysis.
Interpreting Output: Reading the regression output, focusing on adjusted R-squared, significance value (F-test), and P-values.
Coefficient Interpretation: Understanding the impact of each independent variable on the dependent variable, including scaling factors (per 1000 units).
Model Evaluation: Evaluating the model based on significance values and understanding the implications of unexpected results (e.g., stringency index).
Further Analysis: Recognizing the need for additional analysis when encountering unexpected or inconclusive results.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

REST APIs#

REST (Representational State Transfer) APIs are the standard way to build web services that allow different systems to communicate over HTTP. They use standard HTTP methods and JSON for data exchange.

Watch this comprehensive introduction to REST APIs (52 min):

Key Concepts:

HTTP Methods
- GET: Retrieve data
- POST: Create new data
- PUT/PATCH: Update existing data
- DELETE: Remove data
Status Codes
- 2xx: Success (200 OK, 201 Created)
- 4xx: Client errors (400 Bad Request, 404 Not Found)
- 5xx: Server errors (500 Internal Server Error)

Here’s a minimal REST API using FastAPI. Run this server.py script via uv run server.py:

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval Augmented Generation#

Watch the walkthrough above, then dive into the notebook for step-by-step practice.

You will learn to implement Retrieval Augmented Generation (RAG) to enhance language models’ responses by incorporating relevant context, covering:

LLM Context Limitations: Understanding the constraints of context windows in large language models.
Retrieval Augmented Generation: The technique of retrieving and using relevant documents to enhance model responses.
Embeddings: How to convert text into numerical representations that are used for similarity calculations.
Similarity Search: Finding the most relevant documents by calculating cosine similarity between embeddings.
OpenAI API Integration: Using the OpenAI API to generate responses based on the most relevant documents.
Tourist Recommendation Bot: Building a bot that recommends tourist attractions based on user interests using embeddings.
Next Steps for Implementation: Insights into scaling the solution with a vector database, re-rankers, and improved prompts for better accuracy and efficiency.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

RevealJS: Web-Based Presentations#

RevealJS is a powerful HTML presentation framework that turns web pages into interactive slideshows. It’s particularly useful for:

Data Presentations: Interactive charts, live data demos
Technical Talks: Code with syntax highlighting, math equations
Web Portfolios: Showcase projects with live demos
Educational Content: Interactive learning materials
Conference Talks: Remote-friendly with speaker notes
Documentation: API and library showcases

This tutorial covers creating interactive presentations with RevealJS.

Mon, 01 Jan 0001 00:00:00 +0000

Scheduled Scraping with GitHub Actions#

GitHub Actions provides an excellent platform for running web scrapers on a schedule. This tutorial shows how to automate data collection from websites using GitHub Actions workflows.

Key Concepts#

Scheduling: Use cron syntax to run scrapers at specific times
Dependencies: Install required packages like httpx, lxml
Data Storage: Save scraped data to files and commit back to the repository
Error Handling: Implement robust error handling for network issues and HTML parsing
Rate Limiting: Respect website terms of service and implement delays between requests

Here’s a sample scrape.py that scrapes the IMDb Top 250 movies using httpx and lxml:

Mon, 01 Jan 0001 00:00:00 +0000

Scraping emarketer#

In this live scraping session, we explore a real-life scenario where Straive had to scrape data from emarketer.com for a demo. This is a fairly realistic and representative way of how one might go about scraping a website.

You’ll learn:

Scraping: How to extract data from web pages, including constructing URLs, fetching page content, and parsing HTML using packages like lxml and httpx .
Caching: Implementing a caching strategy to avoid redundant data fetching for efficiency and reliability.
Error Handling and Debugging: Practical tips for troubleshooting, such as using liberal print statements, breakpoints for in-depth debugging, and the concept of “rubber duck debugging” to clarify problems.
LLMs: Benefits of Gemini / ChatGPT for code suggestions and troubleshooting.
Real-World Application: How quick proofs of concept to showcase capabilities to clients, emphasizing practice over theory.

Mon, 01 Jan 0001 00:00:00 +0000

Scraping IMDb with JavaScript#

You’ll learn how to scrape the IMDb Top 250 movies directly in the browser using JavaScript on the Chrome DevTools, covering:

Access Developer Tools: Use F12 or right-click > Inspect to open developer tools in Chrome or Edge.
Inspect Elements: Identify and inspect HTML elements using the Elements tab.
Query Selectors: Use document.querySelectorAll and document.querySelector to find elements by CSS class.
Extract Text Content: Retrieve text content from elements using JavaScript.
Functional Programming: Apply map and arrow functions for concise data processing.
Data Structuring: Collect and format data into an array of arrays.
Copying Data: Use the copy function to transfer data to the clipboard.
Convert to Spreadsheet: Use online tools to convert JSON data to CSV or Excel format.
Text Manipulation: Perform text splitting and cleaning in Excel for final data formatting.

Here are links and references:

Mon, 01 Jan 0001 00:00:00 +0000

Scraping: Live Sessions#

Fundamentals of web scraping with urllib and BeautifulSoup

Intermediate web scraping use of cookies

XML intro and scraping

Mon, 01 Jan 0001 00:00:00 +0000

Scraping PDFs with Tabula#

You’ll learn how to scrape tables from PDFs using the tabula Python library, covering:

Import Libraries: Use Beautiful Soup for URL parsing and Tabula for extracting tables from PDFs.
Specify Save Location: Mount Google Drive to save scraped PDFs.
Identify PDF URLs: Parse the given URL to identify and select all PDF links.
Download PDFs: Loop through identified links, saving each PDF to the specified location.
Extract Tables: Use Tabula to read tabular content from the downloaded PDFs.
Control Extraction Area: Specify page and area coordinates to accurately extract tables, avoiding extraneous text.
Save Extracted Data: Convert the extracted table data into structured formats like CSV for further analysis.

Here are links and references:

Mon, 01 Jan 0001 00:00:00 +0000

Scraping with Excel#

You’ll learn how to import tables on the web using Excel , covering:

Data Import from Web: Use the query feature in Excel to scrape data from websites.
Establishing Web Connections: Connect Excel to a web page using a URL.
Using Query Editor: Navigate the query editor to view and manage web data tables.
Loading Data: Load data from the web into Excel for further manipulation.
Data Transformation: Remove unnecessary columns and transform data as needed.
Applying Transformations: Track applied steps in the sequence for reproducibility.
Refreshing Data: Refresh the imported data to get the latest updates from the web.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Scraping with Google Sheets#

You’ll learn how to import tables on the web using Google Sheets’s =IMPORTHTML() formula , covering:

Import HTML Formula: Use =IMPORTHTML(URL, “query”, index) to fetch tables or lists from a web page.
Granting Access: Allow access for formulas to fetch data from external sources.
Checking Imported Data: Verify if the imported table matches the data on the web page.
Handling Errors: Understand common issues and how to resolve them.
Sorting Data: Copy imported data as values and sort it within Google Sheets.
Freezing Rows: Use frozen rows to maintain headers while sorting.
Live Formulas: Learn how web data updates automatically when the source changes.
Other Import Functions: IMPORTXML, IMPORTFEED, IMPORTRANGE, and IMPORTDATA for advanced data fetching options.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Splitting Text in Excel#

You’ll learn how to transform a single-column data set into multiple, organized columns based on specific delimiters using the “Text to Columns” feature.

Here are links used in the video:

US Senate Legislation - Votes

Mon, 01 Jan 0001 00:00:00 +0000

Spreadsheet: Excel, Google Sheets#

You’ll use spreadsheets for data cleaning and exploration. The most popular spreadsheet program is Microsoft Excel followed by Google Sheets .

You may be already familiar with these. If not, make sure to learn the basics of both.

Go through the Microsoft Excel video training and make sure you cover:

Watch this video for an introduction to Google Sheets (49 min):

Mon, 01 Jan 0001 00:00:00 +0000

Database: SQLite#

Relational databases are used to store data in a structured way. You’ll often access databases created by others for analysis.

PostgreSQL, MySQL, MS SQL, Oracle, etc. are popular databases. But the most installed database is SQLite . It’s embedded into many devices and apps (e.g. your phone, browser, etc.). It’s lightweight but very scalable and powerful.

Watch these introductory videos to understand SQLite and how it’s used in Python (34 min):

Mon, 01 Jan 0001 00:00:00 +0000

System Requirements#

This course requires the following:

Software to be installed on your computer.
System permissions to be granted.
Access to the Websites listed below.

Software#

Core tools:

Visual Studio Code
uv for Python
NodeJS for JavaScript
Docker or Podman
GitHub Copilot

Module-specific tools:

System permissions#

Open Developer Tools on Chrome / Edge - to use F12 or Inspect in browser
Open port 8000-9000 for local development server
pip install and npm install to install packages

Websites#

accounts.google.com
agents.md
ai.google.dev
aipipe.org
aiproxy.sanand.workers.dev
aircloak.com
airflow.apache.org
aistudio.google.com
anthropic.com
api-atlas.nomic.ai
api.example.com
apify.com
api.jina.ai
api.openai.com
api.open-notify.org
app.example.com
app.flourish.studio
apple.com
appsource.microsoft.com
arxiv.org
aspiegel.com
atlas.nomic.ai
azure.microsoft.com
base64decode.org
bbc.com
beautiful-soup-4.readthedocs.io
bing.com
blog.gramener.com
bolt.new
calendar.google.com
cb-prod.seek.study.iitm.ac.in
cdn.jsdelivr.net
cdn.openai.com
chat.deepseek.com
chatgpt.com
chat.openai.com
chat.qwen.ai
claude.ai
cli.github.com
cline.bot
cloud.google.com
cmdlinetips.com
codeium.com
code.visualstudio.com
coffee-reviews.prayashm.com
colab.research.google.com
console.cloud.google.com
context7.com
continue.dev
cors-test.codehappy.devs
cran.r-project.org
cursor.com
cyberscoop.com
dashboard.ngrok.com
dataforseo.com
data.gov
data.gov.in
data.gov.ru
datameet.org
datasetsearch.research.google.com
datasets.imdbws.com
datasette.io
dbeaver.io
dbfopener.com
deepwiki.com
desktop.github.com
devanshikat.github.io
developer.chrome.com
developer.imdb.com
developer.mozilla.org
developers.googleblog.com
developers.google.com
dev.mysql.com
discourse.onlinedegree.iitm.ac.in
diva-gis.org
dl.acm.org
docker.com
docs.anthropic.com
docs.astral.sh
docs.cursor.com
docs.docker.com
docs.getdbt.com
docs.github.com
docs.google.com
docs.klarna.com
docs.kumu.io
docs.lovable.dev
docs.nomic.ai
docs.npmjs.com
docs.python.org
docs.python-requests.org
docs.replit.com
docs.sqlalchemy.org
docs.together.ai
docs.windsurf.com
dopiaza.org
drive.google.com
drive.usercontent.google.com
duckdb.org
elastic.co
en.wikipedia.org
evanhahn.github.io
example.com
exam.sanand.workers.dev
explainxkcd.com
fastapi.tiangolo.com
fastapi-users.github.io
ffmpeg.lav.io
ffmpeg.org
flourish.studio
flukeout.github.io
fmwconcepts.com
forms.gle
freecodecamp.org
gemini.google
gemini.google.com
generativelanguage.googleapis.com
geopy.readthedocs.io
getdbt.com
github.com
github.github.com
gitingest.com
gitlab.com
gitlens.amod.io
git-lfs.github.com
git-scm.com
gnu.org
googlecloudcommunity.com
google.com
gramener.com
groups.google.com
heroku.com
howstat.com
httpbin.org
httpie.io
httrack.com
hub.docker.com
hub.getdbt.com
huggingface.co
imagemagick.org
imageoptim.com
imdb.com
imgs.xkcd.com
img.youtube.com
i.ytimg.com
jamendo.com
jeroenjanssens.com
jina.ai
jmespath.org
jqlang.org
jqplay.org
jsoneditoronline.org
json-generator.com
jsonlines.org
jsonlint.com
jsonpath.com
jsonpathfinder.com
json-schema.org
jsonschemavalidator.net
judgments.ecourts.gov.in
kaggle.com
keras.io
khanacademy.org
kumu.io
learn.microsoft.com
linkedin.com
llm.datasette.io
llmstxt.org
localhost
locator-service.api.bbci.co.uk
lovable.dev
lxml.de
lynx.invisible-island.net
macstories.net
magickstudio.imagemagick.org
mail.google.com
makersuite.google.com
mapshaper.org
marimo.app
marimo.io
marketplace.visualstudio.com
marp.app
matplotlib.org
medium.com
microsoft.com
mistral.ai
modelcontextprotocol.io
mongodb.com
mupdf.com
nchandrasekharr.github.io
news.ycombinator.com
ngrok.com
nodejs.org
nominatim.org
notebooklm.google.com
npmjs.com
numpy.org
observablehq.com
obsproject.com
ollama.com
openai.com
openpolicyagent.org
openrefine.org
openrouter.ai
ourworldindata.org
packaging.python.org
pages.github.com
pandas.pydata.org
pandoc.org
parquet.apache.org
pillow.readthedocs.io
platform.openai.com
playwright.dev
pngquant.org
podman.io
pokeapi.co
posit.co
postgresql.org
postman.com
pre-commit.com
projector.tensorflow.org
promptfoo.dev
pydantic-docs.helpmanual.io
pymotw.com
pymupdf.readthedocs.io
pypi.org
python-httpx.org
python-visualization.github.io
qgis.org
quotes.toscrape.com
rasagy.in
rattle.togaware.com
raw.githubusercontent.com
rawgraphs.io
realpython.com
redis.io
relational-data.org
revealjs.com
rishabhmakes.github.io
roocode.com
sanand0.github.io
s-anand.net
scikit-learn.org
scikit-network.readthedocs.io
screentogif.com
seaborn.pydata.org
seek.onlinedegree.iitm.ac.in
senate.gov
shapechef.com
sharp.pixelplumbing.com
simonwillison.net
soundcloud.com
sourceforge.net
sqlite.org
sqlitestudio.pl
sqlite-utils.datasette.io
sqlmodel.tiangolo.com
sqlzoo.net
squoosh.app
stackoverflow.blog
stackoverflow.com
statsmodels.org
stats.stackexchange.com
stedolan.github.io
story-b0f1c.web.app
storytellingwithdata.com
study.iitm.ac.in
superlinked.com
support.anthropic.com
support.apple.com
support.google.com
support.microsoft.com
support.office.com
survey.stackoverflow.co
swagger.io
tableau.com
tabula-py.readthedocs.io
tds.s-anand.net
textblob.readthedocs.io
texttospeech.googleapis.com
thoughtworks.com
timeanddate.com
tools.s-anand.net
tracebit.com
twitter.com
typesense.org
unstructured.io
upload.wikimedia.org
url.com
us-central1-aiplatform.googleapis.com
vercel.com
w3m.sourceforge.net
w3.org
weather-broker-cdn.api.bbci.co.uk
web.archive.org
webmaster.petalsearch.com
whitehouse.gov
whois.com
wikipedia.readthedocs.io
windsurf.com
wordlift.io
xpdfreader.com
yoast.com
your-app.vercel.app
youtu.be
youtube.com
youtubetranscript.com

Mon, 01 Jan 0001 00:00:00 +0000

TDS GPT Reviewer#

After the later parts of this course’s contents were written, we ran it through a Technical Content Reviewer GPT .

Take a look at the GPT’s instructions. These were generated by the OpenAI Prompt Generation tool.

As a **Content Reviewer** for a high school–level course on Tools in Data Science, your job is to evaluate provided content (such as text, code snippets, or references) with a focus on correctness, clarity, and conciseness, and offer actionable feedback for improvement.

1. **Check for Correctness and Consistency**
 - Verify technical and factual accuracy.
 - Ensure internal consistency without contradictions.
2. **Check for Clarity and Approachability**
 - Ensure content is understandable for a high school student with limited prior knowledge.
 - Identify and simplify jargon or advanced concepts.
3. **Check for Conciseness**
 - Assess if content is direct and free of unnecessary verbosity.
 - Identify areas for streamlining to enhance readability.
4. **Provide Feedback for Improvement**
 - Offer actionable suggestions for fixing, clarifying, or reorganizing content.
 - Propose alternative phrasing if text is vague, complex, or verbose.

# Steps

1. Carefully read the entire content before forming conclusions.
2. List factual inconsistencies or missing details causing confusion.
3. Suggest simpler terms or analogies for complex language.
4. Point out unnecessary repetition or filler text.
5. Provide direct examples of how to improve the highlighted issues.

# Output Format

Respond using **Markdown** with the following structure:

1. **Summary of Findings**
 - A concise paragraph outlining overall strengths and weaknesses.
2. **Detailed Review**
 - **Correctness and Consistency**: Note factual errors or inconsistencies, suggesting corrections.
 - **Clarity and Approachability**: Identify overly advanced or unclear sections, offering simpler alternatives.
 - **Conciseness**: Highlight long or repetitive sections with suggestions for tightening the text.
3. **Actionable Improvement Suggestions**
 - Provide specific sentences, bullet points, or rewritten examples to illustrate improvements.

# Notes

- Maintain a constructive review tone, not content generation.
- Even if content is perfect, confirm with suggestions for minor improvements (e.g., adding an example or clarifying a subtle point).

Content creation prompts#

In addition, here are a few prompts used to create the content:

Mon, 01 Jan 0001 00:00:00 +0000

TDS TA Instructions#

The TDS TA is a virtual assistant that helps you with your doubts.

It has been trained on course content created as follows:

# Clone the course repository
git clone https://github.com/sanand0/tools-in-data-science-public.git
cd tools-in-data-science-public

# Create a prompt file for the TA
PYTHONUTF8=1 uvx files-to-prompt --cxml *.md -o tds-content.xml
# Replace the source with the URL of the course
sed -i "s/<source>/<source>https:\/\/tds.s-anand.net\/#\//g" tds-content.xml

Additionally, we visit each of the evaluation links on https://exam.sanand.workers.dev/ , copy it as Markdown , and add it to the content, called ga1.md, ga2.md, etc.

Mon, 01 Jan 0001 00:00:00 +0000

Topic Modeling#

You’ll learn to use text embeddings to find text similarity and use that to create topics automatically from text, covering:

Embeddings: How large language models convert text into numerical representations.
Similarity Measurement: Understanding how similar embeddings indicate similar meanings.
Embedding Visualization: Using tools like Tensorflow Projector to visualize embedding spaces.
Embedding Applications: Using embeddings for tasks like classification and clustering.
OpenAI Embeddings: Using OpenAI’s API to generate embeddings for text.
Model Comparison: Exploring different embedding models and their strengths and weaknesses.
Cosine Similarity: Calculating cosine similarity between embeddings for more reliable similarity measures.
Embedding Cost: Understanding the cost of generating embeddings using OpenAI’s API.
Embedding Range: Understanding the range of values in embeddings and their significance.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Transforming Images#

Image Processing with PIL (Pillow)#

Pillow is Python’s leading library for image processing, offering powerful tools for editing, analyzing, and generating images. It handles various formats (PNG, JPEG, GIF, etc.) and provides operations from basic resizing to complex filters.

Here’s a minimal example showing common operations:

# /// script
# requires-python = ">=3.11"
# dependencies = ["Pillow"]
# ///

from PIL import Image, ImageEnhance, ImageFilter


async def process_image(path: str) -> Image.Image:
 """Process an image with basic enhancements."""
 with Image.open(path) as img:
 # Convert to RGB to ensure compatibility
 img = img.convert("RGB")

 # Resize maintaining aspect ratio
 img.thumbnail((800, 800))

 # Apply enhancements
 img = ImageEnhance.Contrast(img).enhance(1.2)

 return img.filter(ImageFilter.SHARPEN)


if __name__ == "__main__":
 import asyncio

 img = asyncio.run(process_image("input.jpg"))
 img.save("output.jpg", quality=85)

Key features and techniques you’ll learn:

Mon, 01 Jan 0001 00:00:00 +0000

Unicode#

Ever noticed when you copy-paste some text and get garbage symbols? Or see garbage when you load a CSV file? This video explains why. It covers how computers store text (called character encoding) and why it sometimes goes wonky.

Learn about ASCII (the original 7-bit encoding system that could only handle 128 characters), why that wasn’t enough for global languages, and how modern solutions like Unicode save the day by letting us use any character from any language.

Mon, 01 Jan 0001 00:00:00 +0000

Python tools: uv#

Install uv .

uv is a fast Python package and project manager that’s becoming the standard for running Python scripts. It replaces tools like pip, conda, pipx, poetry, pyenv, twine, and virtualenv into one, enabling:

Python Version Management: uv installs and manages multiple Python versions, allowing developers to specify and switch between versions seamlessly.
Virtual Environment Handling: It automates the creation and management of virtual environments, ensuring isolated and consistent development spaces for different projects.
Dependency Management: With support for the pyproject.toml format, uv enables precise specification of project dependencies. It maintains a universal lockfile, uv.lock, to ensure reproducible installations across different systems.
Project Execution: The uv run command allows for the execution of scripts and applications within the managed environment, streamlining development workflows.

Here are some commonly used commands:

Mon, 01 Jan 0001 00:00:00 +0000

Vector Databases#

Vector databases are specialized databases that store and search vector embeddings efficiently.

Use vector databases when your embeddings exceed available memory or when you want it run fast at scale. (This is important. If your code runs fast and fits in memory, you DON’T need a vector database. You can can use numpy for these tasks.)

Vector databases are an evolving space.

The first generation of vector databases were written in C and typically used an algorithm called HNSW (a way to approximately find the nearest neighbor). Some popular ones are:

Mon, 01 Jan 0001 00:00:00 +0000

Serverless hosting: Vercel#

Serverless platforms let you rent a single function instead of an entire machine. They’re perfect for small web tools that don’t need to run all the time. Here are some common real-life uses:

Mon, 01 Jan 0001 00:00:00 +0000

Vibe Analysis#

Vibe analysis is analyzing data as if the analysis itself doesn’t exist—you focus only on business outcomes, skipping intermediate steps.

This approach emerged from vibe coding, where you code as if code doesn’t exist. With vibe analysis, you give an LLM full context, state your goal, and review only the answers. The LLM handles exploration, cleaning, modeling, visualization, and deployment.

Watch these comprehensive talks on vibe analysis and the changing role of data scientists:

Mon, 01 Jan 0001 00:00:00 +0000

Vibe-coding#

The rise of AI coding tools has introduced a new development philosophy: vibe-coding .

Vibe coding is different from AI coding. It’s “fully giving in to the vibes…” and “forgetting that the code even exists”, i.e., accept-all edits, don’t hand-edit diffs, etc. The goal is to get to a working prototype as fast as possible.

Ship scrappy first drafts fast: Focus on getting a working version quickly rather than polished code. AI can generate functional prototypes in minutes, allowing you to validate ideas before investing in refinement.

Mon, 01 Jan 0001 00:00:00 +0000

Vision Models#

You’ll learn how to use LLMs to interpret images and extract useful information, covering:

Setting Up Vision Models: Integrate vision capabilities with LLMs using APIs like OpenAI’s Chat Completion.
Sending Image URLs for Analysis: Pass URLs or base64-encoded images to LLMs for processing.
Reading Image Responses: Get detailed textual descriptions of images, from scenic landscapes to specific objects like cricketers or bank statements.
Extracting Data from Images: Convert extracted image data to various formats like Markdown tables or JSON arrays.
Handling Model Hallucinations: Address inaccuracies in extraction results, understanding how different prompts can affect output quality.
Cost Management for Vision Models: Adjust detail settings (e.g., “detail: low”) to balance cost and output precision.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Animated Data with Flourish#

You’ll learn how to create and customize animations in Flourish, covering:

Core Animation Principles: Understand how to use animation to both engage and inform, and learn the concept of “object constancy” to turn data visualizations into compelling data stories.
Configuring Race Charts: Master the animation controls for Line Chart Races and Bar Chart Races, including setting durations for timelines, transitions, and individual stages.
Morphing Between Chart Types: Learn to use the Line Bar Pie template to create seamless morphing transitions between different chart types (e.g., line to streamgraph) within a Flourish story.
Animating Scatter Plots: Discover how to set up and control animations in scatter plots, including using the “Animation stagger” setting to create dynamic, cascading dot movements.
Controlling Animation Logic: Understand how to manage animations for both related and unrelated datasets by toggling the “Only animate series with same name” option.
Animation in Other Templates: Get an overview of the animation settings available in various other templates, including Survey, Heatmap, Spider, and Hierarchy charts.
Advanced Data Explorer Animations: Learn about the powerful Data Explorer template (an Enterprise feature) and its ability to create advanced animations, such as morphing dots from a survey layout into their corresponding locations on a projection map.

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Animated Data with PowerPoint#

You’ll learn how to create an animated bar chart race in PowerPoint, covering:

Using the Morph Transition: How to leverage PowerPoint’s Morph feature to animate the resizing and reordering of bars between slides, which is the core technique for the race effect.
Manual Bar Creation: The process of using individual shapes (rectangles) instead of a standard chart and how to size them accurately to represent your data values.
Adding and Syncing Audio: How to record a voiceover for each slide, add background music that plays across the entire presentation, and sync them with the animations.
Automating Playback: Setting up automatic slide transitions to create a self-playing, video-like presentation that advances after the narration for each slide is complete.
Exporting as a Video: The steps to save your final animation as an MPEG-4 video file directly from PowerPoint and the suggestion of alternative methods like OBS for higher quality.
Prototyping Data Stories: Understanding the manual effort required for this method and how it allows for careful control when prototyping and crafting a specific visual narrative.

How to make bar chart race in PowerPoint#

Source: How to make a bar chart race in PowerPoint

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Forecasts with Excel#

Excel File used in the video

You’ll learn to perform exploratory data visualization on time-series financial data using Excel, covering:

Visualizing Trends with Sparklines: Use PivotTables and Sparklines to create compact, in-cell charts that reveal the performance trend of various financial securities (currencies, indices, commodities) over time.
Forecasting and Measuring Volatility: Learn to calculate key metrics like the average value, use the GROWTH function to forecast the next day’s performance, and measure volatility using standard deviation.
Analyzing Relationships with Scatter Plots: Create scatter plots to visually investigate how two different securities move in relation to each other.
Quantifying Correlation with R-Squared: Add trendlines and display the R-squared value on a scatter plot to quantitatively measure how much of the movement in one variable is explained by another.
Using the Data Analysis ToolPak: Learn how to enable and use Excel’s powerful Data Analysis ToolPak to perform more advanced statistical analysis.
Creating a Correlation Matrix: Use the ToolPak to efficiently generate a complete correlation matrix, showing the correlation coefficient for every pair of securities in the dataset.
Interpreting a Correlation Matrix with Conditional Formatting: Apply color scales to the matrix to instantly identify strong positive correlations, negative correlations, and outliers.
Discovering Insights and Forming Hypotheses: Learn how to use these visualizations to spot patterns and form data-driven hypotheses, such as the high correlation between the currencies of neighboring countries.

Here’s another version of this analysis, delivered as part of a talk:

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Machine Learning#

You’ll learn about improving customer retention, understanding black box models, and using clustering for market segmentation:

Churn Reduction: Use decision trees to identify customers likely to leave.
Cost Efficiency: Compare customer acquisition vs. retention costs.
Model Improvement: Apply SVMs and neural networks for better accuracy.
Project Challenges: Understand issues with black box models in implementation.
K-Means Clustering: Segment markets using demographic data.
Data Visualization: Interpret clustering results using maps and charts.
Correlation Analysis: Identify relationships between currency exchange rates.
Tool Proficiency: Utilize Excel, Python, and JavaScript for analysis and communication.
Practical Application: Tailor marketing strategies based on cluster characteristics.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Network Data with Kumu#

You’ll learn about visualizing and analyzing relationships and networks using Kumu, covering:

Understanding Kumu: What Kumu is and its primary function as a tool to visualize complex relationships within data. You’ll learn that it’s applicable beyond social analysis to any scenario involving relationships between entities.
Social Network Analysis: How Kumu facilitates social network analysis, helping to understand how different people and communities are connected, and identifying common interests.
Data Preparation for Kumu: The process of preparing raw data, specifically IMDb actor data for Indian movies, to be uploaded to Kumu. This includes filtering for only movies and Indian movies.
Creating Actor Networks: How to construct an actor collaboration matrix where each element denotes the number of movies two actors have acted in together. This involves a method using matrix multiplication of a movie-actor matrix with its transpose.
Optimizing Sparse Matrices: Understanding that actor collaboration matrices are often sparse (contain many zero entries) and how to make computations fast and memory-efficient using the compressed sparse row (CSR) format from the scipy library in Python.
Preparing Data for Kumu Upload: How to convert the processed matrix data into the “from node to node” format, along with the strength of the connection (number of shared movies), which is required for Kumu.
Filtering Data Effectively:
- Filtering by Year: How to filter movie data by release year (e.g., movies released after 1950) by converting the ‘start year’ column to an integer data type, and troubleshooting common issues like newline characters (/n) within string data.
- Filtering by Language/Region: How to filter for specific regions or languages, such as Indian movies, by applying language options within data processing functions or by filtering a dedicated region data frame.
- Filtering Actor Pairs: How to reduce the data size by filtering for actors who have acted in a minimum number of movies and for actor pairs who have acted together in a minimum number of movies.
Visualizing and Analyzing Networks in Kumu: How the prepared data creates a network of actors in Kumu, allowing you to observe clusters and understand direct and indirect connections.
Exploring Network Connections: How to search for specific actors (e.g., Mohanlal) and examine their direct connections (e.g., Mohanlal and Saikumar acted in 8 movies) and indirect connections within the network.
Introduction to Community Detection: A brief mention of community detection as a method to identify groups within the network and understand their interconnections, to be explored in other tutorials.
General Data Science Practices: The importance of using resources like Google and documentation for problem-solving, even for seemingly simple tasks, and the necessity of ensuring correct data types for operations.

Mon, 01 Jan 0001 00:00:00 +0000

Editor: VS Code#

Your editor is the most important tool in your arsenal. That’s where you’ll spend most of your time. Make sure you’re comfortable with it.

Visual Studio Code is, by far, the most popular code editor today. According to the 2025 StackOverflow Survey ~75% of developers use it. We recommend you learn it well. Even if you use another editor, you’ll be working with others who use it, and it’s a good idea to have some exposure.

Mon, 01 Jan 0001 00:00:00 +0000

Web Scraping with Playwright in Python#

Scrape JavaScript‑heavy sites effortlessly with Playwright.

(youtube.com )

Playwright offers:

JavaScript rendering: Executes page scripts so you scrape only after content appears. (playwright.dev )
Headless & headed modes: Run without UI or in a real browser for debugging. (playwright.dev )
Auto‑waiting & retry: Built‑in locators reduce flakiness. (playwright.dev )
Multi‑browser support: Chromium, Firefox, WebKit—all from one API. (playwright.dev )

Example: Scraping a JS‑Rendered Site#

We’ll scrape Quotes to Scrape (JS) —a site that loads quotes via JavaScript, so a simple requests call gets only an empty shell (quotes.toscrape.com ). Playwright runs the scripts and gives us the real content:

Mon, 01 Jan 0001 00:00:00 +0000

Wikipedia Data with Python#

You’ll learn how to scrape data from Wikipedia using the wikipedia Python library, covering:

Installing and Importing: Use pip install to get the Wikipedia library and import it with import wikipedia as wk.
Keyword Search: Use the search function to find Wikipedia pages containing a specific keyword, limiting results with the results argument.
Fetching Summaries: Use the summary function to get a concise summary of a Wikipedia page, limiting sentences with the sentences argument.
Retrieving Full Pages: Use the page function to obtain the full content of a Wikipedia page, including sections and references.
Accessing URLs: Retrieve the URL of a Wikipedia page using the url attribute of the page object.
Extracting References: Use the references attribute to get all reference links from a Wikipedia page.
Fetching Images: Access all images on a Wikipedia page via the images attribute, which returns a list of image URLs.
Extracting Tables: Use the pandas.read_html function to extract tables from the HTML content of a Wikipedia page, being mindful of table indices.

Here are links and references:

Tools in Data Science

Actor Network Visualization#

AI Coding in the CLI#

Claude Code CLI#

AI Coding Context Engineering#

AI Coding in IDEs#

Copy-paste workflows#

AI Coding Online#

Artifacts: Inline AI coding sandboxes#

AI Coding Strategies#

AI Coding Tests#

Build evaluation loops#

AI Coding Tools#

AI Coding#

Archived content#

Videos#

IITM BS Degree - Diploma Level Orientation#

Tools in Data Science Orientation#

Tools in Data Science Live Sessions#

Tools in Data Science Playlist#

Optional: Parse & clean PDF files with Tabula#

Base 64 Encoding#

Terminal: Bash#

BBC Weather location ID with Python#

Cleaning Data with OpenRefine#

Notebooks: Google Colab#

Converting HTML to Markdown#

Converting PDFs to Markdown#

PyMuPDF4LLM#

Correlation with Excel#

CORS: Cross-Origin Resource Sharing#

Crawling with the CLI#

CSS Selectors#

Data Aggregation in Excel#

Data Analysis with Datasette#

Data Analysis with DuckDB#

Data Analysis with Python#

Data Analysis with SQL#

Data analysis#

Data Cleansing in Excel#

Data Preparation in DuckDB#

Create a Sample Dataset#

Data Preparation in the Editor#

Data Preparation in the Shell#

Data Preparation#

Data Sourcing#

Data Storytelling with LLMs#

Prerequisites#

Data Storytelling#

Transcript#

Data Transformation in Excel#

Data Visualization with ChatGPT#

Prerequisites#

Data Visualization with Seaborn#

Data visualization#

Data Transformation with dbt#

Deployment Tools#

Development Tools#

Browser: DevTools#

Containers: Docker, Podman#

Embeddings: OpenAI and Local Models#

Extracting Audio and Transcripts#

Media Processing: FFmpeg#

Web Framework: FastAPI#

May 2025 Feedback#

Insights#

Interesting observations#

Actions#

Fine tuning LLMs: Gemini#

Forecasting with Excel#

Function Calling with OpenAI#

Geospatial Analysis with Excel#

Geospatial Analysis with Python#

Geospatial Analysis with QGIS#

Version Control: Git, GitHub#

CI/CD: GitHub Actions#

IDE: GitHub Codespaces#

AI Editor: GitHub Copilot#

Static hosting: GitHub Pages#

Google Authentication with FastAPI#