PUBLIC CODE DATA SCRAPING

Move public repositoriesinto Code LLM datasets reliably

Synchronize public repositories, archives, history, and large objects with unmetered traffic, 10Gbps+ per-project capacity, and source-specific pools.

Assess high-bandwidth capacity View architecture

Unmetered project traffic10Gbps+ per-project capacityTarget-specific proxy poolsHTTP(S) and SOCKS access

public code pipelinesync ready

Public Code Data123Proxy data network

Public repositoriesREPO

Source archivesARCHIVE

HistoryCOMMITS

Source filesCODE

Repository listMANIFEST

High bandwidthPROXY

Sync workersWORKERS

CAPACITY10Gbps+

TRANSFERUnmetered

PROTOCOLGit / HTTP

10Gbps+Per-project capacity

UnmeteredPredictable transfer cost

Custom poolsTarget-specific routing

24/7Enterprise support

Public Code Data

Bottlenecks Workloads Architecture Proxy strategy Validation Developers FAQ

01 / Workload

Repository count andhistory depth drive transfer cost

Discovery, archives, history, and large objects have different transfer patterns and should use separate queues.

Repositories and branches

Track public project identity and default branches.

Archives and history

Choose current source or full object history.

Incremental sync

Transfer only new or changed objects.

Source and license records

Preserve provenance and public access context.

02 / Scenarios

For large repository setsand continuous incremental sync

Separate network transfer from parsing, license processing, and deduplication.

Public repository mirrors

Synchronize repositories and source archives.

Focus: repository success

Historical object research

Preserve commits, trees, and relationships.

Focus: completeness

Incremental code corpora

Schedule only new or changed objects.

Focus: efficiency

03 / Architecture

Separate repository discovery from object synchronization

Build a manifest first, then run recoverable repository or object tasks through the proxy layer.

Build the public manifest

Record source, repository, branch, time, and license.

Choose archive or Git

Match the protocol to history requirements.

Synchronize with recovery

Retry at repository or object granularity.

Parse and govern

Handle code cleaning and licensing downstream.

Data scraping architecture123Proxy network

Repository listMANIFEST

Proxy poolROUTE

Sync workersWORKERS

Code storeOUTPUT

Primary proxyHigh bandwidth

ProtocolsGit / HTTP

Success unitRepository / object

04 / Proxy strategy

Use high-bandwidth proxy IP as the primary transfer layer

Large AI datasets are constrained first by aggregate throughput and transfer cost. Residential proxies are added only when a workload needs explicit geographic identity.

High-bandwidth proxy IP

Unmetered project traffic10Gbps+ per-project capacityTarget-specific proxy poolsHTTP(S) and SOCKS access

View high-bandwidth proxy IP Contact sales

Primary productHigh-bandwidth proxy IPUnmetered project capacity

Capacity10Gbps+Validated with representative targets

AccessHTTP(S) / SOCKSWorks with existing collectors

05 / Validation

Validate repositories and object completeness

A successful clone alone does not prove references, objects, archives, or large files are complete.

Repository success: Planned public repositories completed
Object completeness: References, trees, commits, and files present
Useful transfer: Successfully stored archives and objects
Incremental efficiency: Changed objects relative to transfer
Recovery success: Interrupted tasks completed later
License records: Repositories with source and license metadata

Validate all capacity assumptions with representative public targets, real response sizes, retry behavior, and the required completion window.

06 / Developer access

Use standard proxies with Git and HTTP clients

Proxy routing does not replace repository discovery, license analysis, code parsing, or deduplication.

Standard proxy configurationExplicit timeout and retry policyMeasure useful data throughput

Git / public repositoryIntegration example

git \
  -c http.proxy="http://user:pass@proxy.123proxy.cn:9000" \
  clone \
  --filter=blob:none \
  "https://target.example/public/repository.git"

07 / FAQ

Questions to answer before production

Validate workload boundaries, useful throughput, retry behavior, and dataset completeness with representative public targets.

Is this limited to GitHub?

No. It applies to lawfully accessible public code hosts, archives, and HTTP objects.

Can private repositories be collected?

Only with valid customer authorization and credentials; the service does not bypass access controls.

Why unmetered transfer?

Repository history, archives, large files, and retries create sustained transfer.

Archive or Git clone?

Use archives for current source and Git when history or object relationships are required.

Is license processing included?

No. License analysis, code cleaning, and deduplication remain customer-owned.

Does 10Gbps+ apply to one repository?

No, it is aggregate capacity for one project.

How should incremental sync work?

Persist update times, references, or object IDs and schedule only changes.

What provenance should be retained?

Store source URLs, timestamps, public state, licenses, and processing rules.

Validate synchronization with representative repositories

Share code sources, repository count, history depth, tools, and completion requirements.

Assess high-bandwidth capacity Contact sales