About Market Insights Server
Market Insights Server tracks commodity markets by collecting social, news, and price data, then processing it with Spark and GPT-4. It is aimed at users seeking configurable, text-based market reports across commodity types.
Key features
- 01Collects Reddit, news, and Yahoo Finance data
- 02Processes text with Apache Spark
- 03Generates GPT-4 market insight reports
- 04Finds commodity-related subreddits and keywords
- 05Exports insights as JSON
Use cases
- 01Reviewing sentiment around a commodity
- 02Combining news narratives with live price feeds
- 03Generating reports for energy, metals, or agriculture
MCP support
The project uses the MCP protocol to provide its commodity market insight capabilities to MCP clients.
Configuration
Local server config is the same for every client. Copy it into your client’s MCP config file.
We haven’t collected a package or remote endpoint for this server. Follow the instructions in the project README.
Tools
Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.
Details
From the project READMEMarket Insights Server
A real-time commodity tracking system using Apache Spark, OpenAI GPT, and the MCP protocol to generate actionable market insights.
Features
- Real-time data collection from Reddit, News APIs, and Yahoo Finance
- Scalable processing using Apache Spark (PySpark 3.5.0)
- Natural language insights powered by GPT-4
- Configurable for any commodity market: energy, metals, agriculture, and more
- Built-in dynamic configuration generation and subreddit discovery
- Ready for deployment with error handling, retries, and async collection
Requirements
- Python 3.x
- PySpark 3.5.0
- Spark NLP 4.4.0
- aiohttp
- yfinance
- openai
- beautifulsoup4
Install dependencies:
pip install -r requirements.txt
Usage
python spark_market_insights_server.py --commodity "nickel"
Outputs:
- Cleaned text data from Reddit and news sources
- TF-IDF features
- GPT-4-powered insight report
- JSON export of insights
Architecture
-
Data Collection Layer
- Async scraping of Reddit and news articles
- Yahoo Finance for live price feeds
-
Processing Layer (Apache Spark)
- Tokenization → Stop words removal → TF-IDF vectorization
- Supports Spark NLP pipelines
-
AI Insight Layer
- Uses OpenAI GPT-4 to summarize and synthesize market narratives
-
Configuration Layer
- Automatically identifies relevant subreddits and keywords per commodity
🔍 Example Output
“Nickel prices rose sharply after Indonesia’s new export ban. Reddit sentiment is bullish, with posts anticipating supply constraints. Market data shows correlated uptick in EV-related equities like NIO and LI.”
🛠 Troubleshooting
| Problem | Fix |
|---|---|
| Spark stage stuck | Check memory settings, repartition input |
| API returns 429 | Add backoff/retry logic, rotate API keys |
| GPT returns empty | Use latest models, tune prompt |
Coming Soon
- Youtube videos to be analyzed as well for the commodities
- Public dashboard on Smithery.ai
Contributing
Have a new data source or insight model? PRs welcome!
License
MIT License
Comments
New to MCP?
From what the protocol is to installing your first server, these answers cover most first questions.
An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.