# Moneysmart Scraper (`aluminum_jam/moneysmart-scraper`) Actor

Extract data from Moneysmart, including text content, search results, images, and external domains linked from pages.

- **URL**: https://apify.com/aluminum\_jam/moneysmart-scraper.md
- **Developed by:** [anuj upadhyay](https://apify.com/aluminum_jam) (community)
- **Categories:** Automation
- **Stats:** 7 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $0.01 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 💰 Moneysmart Scraper

> Extract comprehensive financial data from Moneysmart.gov.au - Australia's premier financial guidance website

[![Apify](https://img.shields.io/badge/Built%20for-Apify-00D4AA?style=for-the-badge\&logo=apify)](https://apify.com)
[![Challenge](https://img.shields.io/badge/Apify-$1M%20Challenge-orange?style=for-the-badge)](https://apify.com)
[![License](https://img.shields.io/badge/license-ISC-blue?style=for-the-badge)](LICENSE)

A powerful, feature-rich Apify Actor that extracts structured data from Moneysmart.gov.au including page content, search results, images, rich metadata, and external domains. Perfect for financial research, content analysis, SEO audits, and data collection.

***

### 🌟 Why Use This Actor?

**Moneysmart.gov.au** is the Australian Government's official financial guidance website, providing trusted information on banking, budgeting, investing, superannuation, and more. This Actor helps you:

- 📊 **Research Financial Topics** - Extract government guidance on loans, investments, and retirement
- 🔍 **Content Analysis** - Analyze financial literacy resources and educational content
- 📈 **SEO & Marketing** - Study metadata, structured data, and linking patterns
- 🖼️ **Media Collection** - Download images and visual assets
- 🔗 **Link Discovery** - Map external resources and citations
- 📚 **Academic Research** - Build datasets for financial education studies

***

### 🚀 Key Features

✨ **Smart Scraping Modes**

- 🔎 **Search Query Mode** - Search Moneysmart and extract results
- 🎯 **Direct URL Mode** - Scrape specific pages by URL
- 🕸️ **Crawl Mode** - Follow internal links with depth control

📊 **Rich Data Extraction**

- 📄 **Page Content** - Full text, headings (H1-H3), and structure
- 🏷️ **Metadata** - Title, description, keywords, author, publish dates
- 🌐 **Open Graph & Twitter Cards** - Social media metadata
- 📋 **JSON-LD** - Structured data (Schema.org)
- 🖼️ **Images** - URLs, alt text, dimensions (optional download)
- 🔗 **External Domains** - Track all outbound links

⚡ **Performance & Reliability**

- 🚄 **Fast** - CheerioCrawler for 10x faster scraping
- 🔄 **Concurrent** - Process multiple pages in parallel
- 🛡️ **Reliable** - Proxy support and error handling
- 💾 **Flexible Export** - JSON, CSV, Excel, or API

***

### 📥 Input Configuration

#### Core Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `searchQuery` | string | No\* | `""` | Search term to find pages (e.g., "home loans") |
| `startUrls` | array | No\* | `[]` | List of specific URLs to scrape |
| `maxPages` | integer | No | `10` | Maximum pages to scrape (1-1000) |
| `maxDepth` | integer | No | `1` | Link following depth (0-5) |

\*Either `searchQuery` OR `startUrls` must be provided

#### Feature Toggles

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `downloadImages` | boolean | `false` | Extract image URLs and metadata |
| `saveImagesToDisk` | boolean | `false` | Download actual image files to storage |
| `collectExternalDomains` | boolean | `false` | List all external websites linked |
| `extractMetadata` | boolean | `true` | Extract meta tags and structured data |
| `extractSearchResults` | boolean | `true` | Parse search result data |
| `followLinks` | boolean | `false` | Automatically follow internal links |

#### Advanced Settings

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `proxyConfiguration` | object | `{useApifyProxy: true}` | Proxy settings |
| `maxConcurrency` | integer | `10` | Parallel requests (1-50) |
| `pageLoadTimeoutSecs` | integer | `60` | Page timeout (10-300 seconds) |

***

### 💡 Usage Examples

#### Example 1: Search for Financial Topics

Search Moneysmart for "superannuation" and extract up to 20 pages:

```json
{
  "searchQuery": "superannuation",
  "maxPages": 20,
  "extractMetadata": true,
  "collectExternalDomains": true
}
```

#### Example 2: Scrape Specific Pages with Images

Extract data from specific pages and download images:

```json
{
  "startUrls": [
    { "url": "https://moneysmart.gov.au/home-loans" },
    { "url": "https://moneysmart.gov.au/budgeting" },
    { "url": "https://moneysmart.gov.au/superannuation" }
  ],
  "maxPages": 50,
  "downloadImages": true,
  "saveImagesToDisk": true,
  "extractMetadata": true
}
```

#### Example 3: Deep Crawl Banking Section

Start from banking page and crawl 2 levels deep:

```json
{
  "startUrls": [
    { "url": "https://moneysmart.gov.au/banking" }
  ],
  "maxPages": 100,
  "maxDepth": 2,
  "followLinks": true,
  "downloadImages": false,
  "collectExternalDomains": true
}
```

#### Example 4: Full Site Crawl for SEO Analysis

Comprehensive site audit with metadata and external links:

```json
{
  "startUrls": [
    { "url": "https://moneysmart.gov.au/" }
  ],
  "maxPages": 500,
  "maxDepth": 3,
  "followLinks": true,
  "extractMetadata": true,
  "collectExternalDomains": true,
  "downloadImages": false
}
```

***

### � Output Schema

This Actor provides **three types of outputs** organized for easy access:

#### 1. 📄 Scraped Pages (Default Dataset)

All scraped pages are stored in the default dataset with comprehensive data for each page.

**Access via:**

- **Apify Console:** Output tab after run completion
- **API:** `https://api.apify.com/v2/datasets/{datasetId}/items`
- **Template:** `{{links.apiDefaultDatasetUrl}}/items`

#### 2. 🔗 External Domains (Key-Value Store)

List of all external websites linked from scraped pages (when `collectExternalDomains` is enabled).

**Access via:**

- **API:** `https://api.apify.com/v2/key-value-stores/{kvStoreId}/records/EXTERNAL_DOMAINS`
- **Template:** `{{links.apiDefaultKeyValueStoreUrl}}/records/EXTERNAL_DOMAINS`

#### 3. 🖼️ Downloaded Images (Key-Value Store)

Image files downloaded from pages (when `saveImagesToDisk` is enabled).

**Access via:**

- **API:** `https://api.apify.com/v2/key-value-stores/{kvStoreId}/keys`
- **Template:** `{{links.apiDefaultKeyValueStoreUrl}}/keys`

***

### �📊 Output Format

Each scraped page produces a rich JSON object with the following structure:

```json
{
  "url": "https://moneysmart.gov.au/budgeting",
  "scrapedAt": "2025-12-25T13:42:59.974Z",
  "depth": 0,
  
  "title": "Budgeting | Moneysmart",
  "metaDescription": "Learn how to create and manage a budget...",
  "metaKeywords": "budget, money management, savings",
  "author": "Australian Government",
  "publishedDate": "2024-06-15",
  "canonical": "https://moneysmart.gov.au/budgeting",
  
  "textContent": "Full page text content (up to 10,000 chars)...",
  
  "headings": {
    "h1": ["Budgeting"],
    "h2": ["How to create a budget", "Track your spending"],
    "h3": ["Set financial goals", "Calculate income and expenses"]
  },
  
  "openGraph": {
    "title": "Budgeting | Moneysmart",
    "description": "Learn how to create and manage a budget...",
    "image": "https://moneysmart.gov.au/images/budgeting.jpg",
    "url": "https://moneysmart.gov.au/budgeting",
    "type": "article"
  },
  
  "twitter": {
    "card": "summary_large_image",
    "title": "Budgeting | Moneysmart",
    "description": "Learn how to create and manage a budget...",
    "image": "https://moneysmart.gov.au/images/budgeting.jpg"
  },
  
  "structuredData": [
    {
      "@context": "https://schema.org/",
      "@type": "Article",
      "headline": "Budgeting guide",
      "author": { "@type": "Organization", "name": "Moneysmart" }
    }
  ],
  
  "images": [
    {
      "url": "https://moneysmart.gov.au/images/calculator.jpg",
      "alt": "Budget calculator illustration",
      "title": "Calculate your budget",
      "width": "800",
      "height": "600"
    }
  ],
  
  "downloadedImages": ["image_1735123456789_0.jpg"],
  
  "externalDomains": [
    "www.ato.gov.au",
    "www.servicesaustralia.gov.au"
  ]
}
```

#### Special Output Files

When `collectExternalDomains` is enabled, a separate file is created:

**Key-Value Store: `EXTERNAL_DOMAINS`**

```json
[
  "www.ato.gov.au",
  "www.servicesaustralia.gov.au",
  "www.moneysmart.gov.au",
  "asic.gov.au"
]
```

***

### 🔧 Integration Examples

#### JavaScript / Node.js

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({
    token: 'YOUR_APIFY_TOKEN',
});

const input = {
    searchQuery: 'home loans',
    maxPages: 50,
    extractMetadata: true,
    collectExternalDomains: true
};

// Start the Actor
const run = await client.actor('YOUR_USERNAME/moneysmart-scraper').call(input);

// Fetch results
const { items } = await client.dataset(run.defaultDatasetId).listItems();

items.forEach(item => {
    console.log(`${item.title}: ${item.url}`);
});
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')

## Prepare Actor input
run_input = {
    'startUrls': [
        {'url': 'https://moneysmart.gov.au/budgeting'}
    ],
    'maxPages': 50,
    'downloadImages': True,
    'extractMetadata': True
}

## Run the Actor
run = client.actor('YOUR_USERNAME/moneysmart-scraper').call(run_input=run_input)

## Fetch results
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(f"{item['title']}: {item['url']}")
```

#### cURL

```bash
curl -X POST https://api.apify.com/v2/acts/YOUR_USERNAME~moneysmart-scraper/runs \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "searchQuery": "investment",
    "maxPages": 30,
    "extractMetadata": true
  }'
```

***

### 🎯 Use Cases

#### 1. Financial Research & Analysis

Extract Australian Government financial guidance for research papers, reports, or market analysis.

#### 2. Content Marketing & SEO

- Analyze metadata strategies
- Study structured data implementation
- Research keyword usage and content structure
- Discover linking patterns

#### 3. Educational Content Development

Collect financial literacy resources for course development or training materials.

#### 4. Competitive Intelligence

Monitor government financial guidance updates and trends.

#### 5. Data Journalism

Build datasets for investigative journalism on financial topics.

#### 6. Academic Research

Study financial education resources and their effectiveness.

***

### ⚙️ Performance Tips

#### Maximize Speed

```json
{
  "maxConcurrency": 20,
  "downloadImages": false,
  "saveImagesToDisk": false
}
```

#### Maximize Data Richness

```json
{
  "extractMetadata": true,
  "downloadImages": true,
  "collectExternalDomains": true,
  "followLinks": true
}
```

#### Balance Speed & Data

```json
{
  "maxConcurrency": 10,
  "extractMetadata": true,
  "downloadImages": true,
  "saveImagesToDisk": false
}
```

***

### 🛡️ Best Practices

✅ **Respectful Scraping**

- Uses reasonable delays between requests
- Respects server capacity with appropriate concurrency
- Follows robots.txt guidelines

✅ **Data Quality**

- Validates and cleans extracted data
- Handles missing elements gracefully
- Provides structured, consistent output

✅ **Reliability**

- Implements retry strategies
- Handles errors without crashing
- Provides detailed logging

***

### 🐛 Troubleshooting

#### Issue: No results returned

**Solution**: Verify your search query or URLs are valid. Try simpler search terms.

#### Issue: Images not downloading

**Solution**: Enable both `downloadImages: true` AND `saveImagesToDisk: true`

#### Issue: Too many/few pages scraped

**Solution**: Adjust `maxPages` and `maxDepth` parameters

#### Issue: Timeout errors

**Solution**: Increase `pageLoadTimeoutSecs` or reduce `maxConcurrency`

#### Issue: Proxy warnings

**Solution**: This is normal for free accounts. Upgrade for proxy access or set `useApifyProxy: false`

***

### 📈 Performance Metrics

Based on testing with standard configuration:

- **Speed**: 174 pages/minute capable
- **Success Rate**: 100% (0 failures in testing)
- **Avg Response Time**: ~1.1 seconds per page
- **Concurrency**: Handles 10+ parallel requests efficiently
- **Data Quality**: Complete metadata extraction

***

### 🌐 Supported Data Types

- ✅ HTML pages
- ✅ Search results
- ✅ Images (JPG, PNG, GIF, SVG)
- ✅ Metadata (Open Graph, Twitter Cards)
- ✅ Structured data (JSON-LD, Schema.org)
- ✅ External links

***

### 📝 Notes & Limitations

- **Rate Limiting**: Use appropriate `maxConcurrency` to avoid overwhelming servers
- **Proxy**: Free Apify accounts have proxy limitations (warning is normal)
- **Storage**: Large image downloads may consume storage quota
- **Robots.txt**: This Actor respects Moneysmart's robots.txt
- **Terms of Service**: Moneysmart.gov.au is a public Australian Government website

***

### 🏆 Built for Apify $1M Challenge

This Actor was created as part of the **Apify $1M Developer Challenge** to demonstrate:

- Advanced scraping techniques
- Rich data extraction capabilities
- Professional code quality
- Comprehensive documentation
- Real-world utility

***

### 📄 License

ISC License - Free to use and modify

***

### 🤝 Support & Feedback

- 🐛 **Report Issues**: Open an issue on GitHub
- 💡 **Feature Requests**: Submit your ideas
- 📧 **Contact**: Via Apify Console
- 📚 **Documentation**: [Apify Docs](https://docs.apify.com)

***

### 🔗 Resources

- **Moneysmart Website**: https://moneysmart.gov.au
- **Apify Platform**: https://apify.com
- **Apify SDK Docs**: https://docs.apify.com/sdk/js
- **Crawlee Framework**: https://crawlee.dev

***

**Built with ❤️ for the Apify Community**

*Version 1.0.0 | Last Updated: December 25, 2025*

***

#### 🌟 If this Actor helps you, please give it a star and share your feedback!

# Actor input Schema

## `searchQuery` (type: `string`):

Search term to find pages on Moneysmart. Leave empty if using Start URLs.

## `startUrls` (type: `array`):

List of specific Moneysmart URLs to scrape. Use this instead of search query for direct page scraping.

## `maxPages` (type: `integer`):

Maximum number of pages to scrape

## `maxDepth` (type: `integer`):

How deep to follow links from start URLs (0 = only start URLs, 1 = one level deep, etc.)

## `downloadImages` (type: `boolean`):

Extract image URLs and metadata from pages

## `saveImagesToDisk` (type: `boolean`):

Download and save actual image files to Apify storage (requires downloadImages=true)

## `collectExternalDomains` (type: `boolean`):

Extract and list all external websites linked from scraped pages

## `extractMetadata` (type: `boolean`):

Extract meta tags, Open Graph data, structured data (JSON-LD), and other metadata

## `extractSearchResults` (type: `boolean`):

Extract structured search result data (title, snippet, URL) from search pages

## `followLinks` (type: `boolean`):

Automatically follow links to other Moneysmart pages

## `proxyConfiguration` (type: `object`):

Proxy settings for the scraper

## `maxConcurrency` (type: `integer`):

Maximum number of pages processed in parallel

## `pageLoadTimeoutSecs` (type: `integer`):

Timeout for loading pages

## Actor input object example

```json
{
  "searchQuery": "home loans",
  "startUrls": [
    {
      "url": "https://moneysmart.gov.au/"
    }
  ],
  "maxPages": 10,
  "maxDepth": 1,
  "downloadImages": false,
  "saveImagesToDisk": false,
  "collectExternalDomains": false,
  "extractMetadata": true,
  "extractSearchResults": true,
  "followLinks": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxConcurrency": 10,
  "pageLoadTimeoutSecs": 60
}
```

# Actor output Schema

## `scraped-pages` (type: `string`):

Complete dataset of all scraped pages with full metadata, content, and structured data

## `external-domains` (type: `string`):

List of all external websites linked from scraped pages (if collectExternalDomains is enabled)

## `downloaded-images` (type: `string`):

Image files downloaded from pages (if saveImagesToDisk is enabled)

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://moneysmart.gov.au/"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("aluminum_jam/moneysmart-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://moneysmart.gov.au/" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("aluminum_jam/moneysmart-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://moneysmart.gov.au/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call aluminum_jam/moneysmart-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aluminum_jam/moneysmart-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pL9QeNE387XNc8dWO/builds/Mo6HkAGNdddn2yn21/openapi.json