Photo by the author Chatgpt
# Entry
Obtaining real data for scientific projects is often the most tough part. The toys are basic to find, but in the case of high -quality data or in real time you usually need to employ API interfaces or the construction of non -standard pipelines to scrape to separate information from the Internet.
In this article I provide my 10 favorite APIs – those that I employ every day to collect data, integrate data and build AI agents. These APIs are organized in five categories, including trusted data repositories, internet scraping and website search, so you can quickly choose the right tool and go faster from data to inspection.
# Basic data repositories
The basic data repository is a platform based on a community in which various organizations and associates of Open Source divide their data sets in a wider world. Thanks to a plain command, you can access these data sets for your project.
// 1. Kaggle API
Kaggle Data sets are extremely popular when working on data science projects. Instead of downloading them by hand, you can create a data pipeline that automatically download a data set, unpack it and load it into your working area.
These data sets are made available by the Open Source community for everyone. To start, generate the API key from the Kaggle account and set it as an environmental variable. Then you can launch the following commands in your terminal. Kaggle also provides SDK Python, which allows basic integration with the code.
kaggle datasets download -d kingabzpro/world-vaccine-progress -p data --unzip// 2. Hugging cli
Similar to kaggle, Hugging It is also a community of data learning and machine learning in which people provide data sets, models and demos. You can easily install the clinging of your face and integrate it with work flows using CLI or Python code. Both options allow you to download data sets without the need for the API key.
The API key is only required if the data set is goal.
hf download kingabzpro/dermatology-qa-firecrawl-dataset# Internet and creeping API interfaces
The network contains a wide range of data. If you cannot find the information you need on the above -mentioned platforms, you may need to scroll your own data by scraping the network or using the API of the network search interface.
// 3. FINERE
Finer Provides the API interface to separate content from websites and transform it into a Markdown format for easier AI integration. It is also equipped with an APi scraping and extraction interface, which is integrated with LLM (Huge Language model) for advanced network scraping options.
This API interface is mandatory. I employ it daily to create data and to integrate them with my AI projects.
curl -s -X POST "https://api.firecrawl.dev/v2/scrape"
-H "Authorization: Bearer $FIRECRAWL_API_KEY"
-H "Content-Type: application/json"
-d '{
"url": "https://abid.work",
"formats": ["markdown", "html"]
}'// 4. Melted
Flux This is a quick interface of the Internet Search API, which provides 1000 search requests per month for free. It is both exact and swift. You can employ it to create data sets, integrate it with AI projects or employ it as a plain API interface for searching for needs.
curl --request POST
--url https://api.tavily.com/search
--header "Authorization: Bearer "
--header "Content-Type: application/json"
--data '{
"query": "who is Leo Messi?",
"auto_parameters": false,
"topic": "general",
"search_depth": "basic",
"chunks_per_source": 3,
"max_results": 1,
"days": 7,
"include_answer": true,
"include_raw_content": true,
"include_images": false,
"include_image_descriptions": false,
"include_favicon": false,
"include_domains": [],
"exclude_domains": [],
"country": null
}' # Georbal and weather API
If you are looking for sets of weather and geopolitative data, you will know that everything is changing. That’s why you need access to these real -time data sets through API.
// 5. OpenWeathermap
Openweathermap It is a service that provides global weather data via API interfaces, including current conditions, forecasts, Nowcasts, historical records and even hyperlox precipitation forecasts per minute.
curl "https://api.openweathermap.org/data/2.5/weather?q=London&appid=YOUR_API_KEY&units=metric"// 6. OpenStreetmap
OpenStreet map provides data maps and API viaduct This is an online read -only database that supports selected non -standard OSM parts and can be asked using the QL viaduct. The following example takes cafe nodes in a compact London limiting box.
curl -G "https://overpass-api.de/api/interpreter"
--data-urlencode 'data=[out:json];node["amenity"="cafe"](51.50,-0.15,51.52,-0.10);out;'
# API of financial market data
The API of financial market data is highly recommended if you are working on a financial project and need real -time data on shares, cryptography and other information and information related to finance.
// 7. Alpha Vantage
curl "https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=IBM&apikey=YOUR_API_KEY"// 8. Yahoo Finance
Yahoo finance It offers free exchange quotes, messages, portfolio tools and international markets range, enabling users to explore a wide range of market data without a direct cost.
import yfinance as yf
print(yf.download("AAPL", period="1y").head())# API of social and social data
If you are working on the project of analyzing text and social conversations from the best social media platforms, these API interfaces provide basic access to real social media data.
// 9. Reddit
Reddit offers a prosperous source of data based on community and Python Reddit API Wrappper (Rights) It facilitates access to the official API Reddit interface in terms of tasks, such as downloading posts, comments and subreddit in Python.
Rights work by sending requests to the API Reddit interface under the hood and is widely used in teaching and research to collect discussion threads for analysis.
import praw
r = praw.Reddit(
client_id="ID",
client_secret="SECRET",
user_agent="myapp:ds-project:v1 (by u/yourname)"
)
print([s.title for s in r.subreddit("Python").hot(limit=5)])
// 10. X
X (previously known as Twitter) provides a programmers’ platform with REST end points for downloading users and content, as well as real -time streaming options. Access generally requires authentication, compliance with limits and rules, and choosing the level of access appropriate to volume and the case of employ.
curl -H "Authorization: Bearer YOUR_BEARER_TOKEN"
"https://api.x.com/2/users/by/username/jack"# Final thoughts
These API interfaces provide free access to data that is often tough to obtain. They significantly improve your ability to collect internet data or improve the efforts of online scraping, enabling the creation of custom data sets.
I highly recommend tabs of this article to visit again when you need high quality data in real time from the Internet. By using these API interfaces, you can unlock valuable insights that will assist in your research and analysis.
Abid Ali Awan (@1abidaliawan) is a certified scientist who loves to build machine learning models. Currently, it focuses on creating content and writing technical blogs on machine learning and data learning technologies. ABID has a master’s degree in technology management and a bachelor’s title in the field of telecommunications engineering. His vision is to build AI with a neural network for students struggling with mental illness.