diff --git a/.gitignore b/.gitignore index 521ef88..6a014ae 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ -*.swp +02packages.details.txt .carton -.tidyall.d/ cpanfile.snapshot local +*.swp +.tidyall.d/ diff --git a/README.md b/README.md index d48c534..2cdb167 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,143 @@ -metacpan-examples -================= - -Sample code to get you up and running quickly. If you are new to MetaCPAN, -please use the scripts/endpoints/author examples to get yourself started. If -you are searching for a specific topic, try scripts/topic as your starting -point. - -The purpose of this repository is to showcase how to use the MetaCPAN API. -There are examples using raw curl as well as the following Perl modules: -MetaCPAN::API, MetaCPAN::API::Tiny, WWW::Mechanize and ElasticSearch. Scripts -are all found in /scripts. You can run the Perl scripts in the usual manner, +# metacpan-examples + +This repository contains sample code to get you up and running quickly with the MetaCPAN API. + +## Please use MetaCPAN::Client + +There are many different ways and clients which you can use to access the API. However, *we strongly encourage you* to start by using the MetaCPAN::Client module. If you have compelling reasons to use a different client, feel free to do so, but please be aware that MetaCPAN::Client is fully supported by MetaCPAN. You will likely save yourself (and us) much debugging time if you begin with this module rather than rolling your own Elasticsearch queries. If MetaCPAN::Client *doesn't* do something which you need it to do, please open a GitHub issue to let us know about it. + +There are some examples to be found here which use `curl`, `WWW::Mechanize`, `Search::Elasticsearch` etc. These are useful for reference, but please do take `MetaCPAN::Client` as your starting point. + +## Getting Started + +Scripts are all found in /scripts. You can run the Perl scripts in the usual manner, but if you want to get up and running quickly, you are encouraged to run your scripts via Carton. The workflow is: cpanm Carton carton install - bin/carton scripts/endpoints/author/1-fetch-single-author-mcpan-api.pl + bin/carton scripts/author/1a-search-authors.pl Or, you can use Carton directly: - carton exec perl -Ilib scripts/endpoints/author/1-fetch-single-author-mcpan-api.pl + carton exec perl -Ilib scripts/author/1a-search-authors.pl -Using the carton wrapper in bin will save you a few keystrokes and will +Using the `bin/carton` will save you a few keystrokes and will automatically add new libs to the path in future, if they are required. Use the workflow you are most comfortable with. Please open issues for examples you would like to see and send pull requests for examples you've already written. + +## Upgrading from v0 + +The MetaCPAN API v1 is now available. v0 will be deprecated after a 6 month window. This window closes on or after June 1, 2017. Here's a guide to converting your scripts. + +## Elasticsearch version + +v1 of the MetaCPAN API uses [Elasticsearch v2.4.0](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/index.html) (v0 was at 0.20.2). There are many breaking changes in this upgrade, since it spans almost 4 years. + +The MetaCPAN API versions will increase with breaking changes, but we will not use the same versioning as Elasticsearch itself. + +### Data Structure Changes + +Elasticsearch 1.x changed the data structure returned when fields are used. +For example before one could get a `ArrayRef[ HashRef [ Str ] ]` where now +that will come in the form of `ArrayRef[ HashRef [ ArrayRef [ Str ] ] ]`. + +Our convenience endpoints revert this behaviour, but if you're crafting your own searches, you should be aware that the structure of your fields data may have changed. + +### Client-specific Changes + +#### MetaCPAN::Client + +v0: + +```perl +use MetaCPAN::Client->new(); +``` + +v1: + +```perl +use MetaCPAN::Client->new( version => 'v1' ); +``` + +#### Search::Elasticsearch + +v0: + +```perl +use Search::Elasticsearch; +my $es = Search::Elasticsearch->new( + cxn_pool => 'Static::NoPing', + nodes => 'api.metacpan.org', + trace_to => 'Stdout', +); +``` + +v1: + +```perl +use Search::Elasticsearch; +my $es = Search::Elasticsearch->new( + cxn_pool => 'Static::NoPing', + nodes => 'https://fastapi.metacpan.org/v1', + send_get_body_as => 'POST', + trace_to => 'Stdout', +); +``` + +* The node URL of the v1 API is https://fastapi.metacpan.org/v1 Note that not + only is the host name new, but we've also switched to https and also added + the version to the path. + +* You'll need to set `send_get_body_as => 'POST'`. This is because v1 does not accept `GET` with a body. + +##### $es->search + +```perl +my $faves = $es->search( + index => 'v0', + type => 'favorite', + body => { + query => { match_all => {} }, + facets => { + dist => { + terms => { field => 'favorite.distribution', size => 10 }, + }, + }, + }, + size => 0, +); +``` + +```perl +my $faves = es()->search( + index => 'cpan', + type => 'favorite', + body => { + aggs => { + dist => { + terms => { field => 'distribution', size => 10 }, + }, + }, + }, +); +``` + +* The index name is now `cpan` +* `match_all` is the default query type, so it's not required to specify it +* `facets` have been replaced by `aggs` +* The data structure returned for `facets` does not mirror the `aggs` return values, so you'll need to rework your logic when checking the return values. +* When searching on a type (`favorite` in this case), do not prefix the key with the type name. So, as seen above, `field => 'favorite.distribution'` is now `field => 'distribution'` + +#### GET via command line or browser + +v0: `http://api.metacpan.org/v0/author/MSTROUT` + +v1: `https://fastapi.metacpan.org/v1/author/MSTROUT` + +* Note the new host name +* Note that we now use HTTPS +* Note that the path begins with v1 rather than v0 diff --git a/cpanfile b/cpanfile index c98f47a..8d18c66 100644 --- a/cpanfile +++ b/cpanfile @@ -5,10 +5,12 @@ requires 'Data::Printer::Filter::URI'; requires 'DateTime'; requires 'HTTP::Tiny::Mech'; requires 'IO::Socket::SSL'; -requires 'JSON'; +requires 'JSON::MaybeXS'; requires 'MetaCPAN::API::Tiny'; -requires 'MetaCPAN::Client', '>= 1.005000'; +requires 'MetaCPAN::Client', '>= 2.000000'; requires 'Search::Elasticsearch'; +requires 'Search::Elasticsearch::Client::2_0::Direct'; +requires 'Sub::Exporter'; requires 'WWW::Mechanize::Cached'; requires 'WWW::Mechanize::Cached::GZip'; diff --git a/lib/MetaCPAN/Util.pm b/lib/MetaCPAN/Util.pm index 280ee5b..e6905f4 100644 --- a/lib/MetaCPAN/Util.pm +++ b/lib/MetaCPAN/Util.pm @@ -3,15 +3,46 @@ package MetaCPAN::Util; use strict; use warnings; +use constant DEBUG => !!$ENV{METACPAN_EXAMPLES_DEBUG}; + use Search::Elasticsearch; use Sub::Exporter -setup => { exports => ['es'] }; sub es { return Search::Elasticsearch->new( - cxn_pool => 'Static::NoPing', - nodes => 'api.metacpan.org', - trace_to => 'Stdout', + client => '2_0::Direct', + cxn_pool => 'Static::NoPing', + nodes => 'https://fastapi.metacpan.org/v1', + send_get_body_as => 'POST', + ( trace_to => 'Stdout' ) x !!(DEBUG), ); } 1; + +=head1 NAME + +MetaCPAN::Util - Utilities for accessing MetaCPAN + +=head1 DESCRIPTION + +Provides shared utility code for examples. + +=head1 FUNCTIONS + +=head2 es + +Returns a L client configured for use with the +MetaCPAN API endpoint. + +=head1 ENVIRONMENT + +=over 4 + +=item C + +When set to C<1>, enables output of verbose debugging information. + +=back + +=cut diff --git a/scripts/author/1-fetch-single-author-curl.sh b/scripts/author/1-fetch-single-author-curl.sh new file mode 100755 index 0000000..34edd3c --- /dev/null +++ b/scripts/author/1-fetch-single-author-curl.sh @@ -0,0 +1,3 @@ +#!/bin/sh + +curl https://fastapi.metacpan.org/v1/author/MSTROUT diff --git a/scripts/author/1-fetch-single-author-es.pl b/scripts/author/1-fetch-single-author-es.pl new file mode 100755 index 0000000..1fb7187 --- /dev/null +++ b/scripts/author/1-fetch-single-author-es.pl @@ -0,0 +1,17 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +use Data::Printer; + +use lib './lib'; +use MetaCPAN::Util qw( es ); + +my $author = es->get( + index => 'cpan', + type => 'author', + id => 'MSTROUT', +); + +p $author; diff --git a/scripts/endpoints/author/1-fetch-single-author-www-mech-cached.pl b/scripts/author/1-fetch-single-author-www-mech-cached.pl similarity index 73% rename from scripts/endpoints/author/1-fetch-single-author-www-mech-cached.pl rename to scripts/author/1-fetch-single-author-www-mech-cached.pl index 12e8db0..dd3342a 100755 --- a/scripts/endpoints/author/1-fetch-single-author-www-mech-cached.pl +++ b/scripts/author/1-fetch-single-author-www-mech-cached.pl @@ -6,6 +6,6 @@ use WWW::Mechanize::Cached::GZip; my $mech = WWW::Mechanize::Cached::GZip->new; -$mech->get('http://api.metacpan.org/v0/author/MSTROUT'); +$mech->get('https://fastapi.metacpan.org/v1/author/MSTROUT'); say $mech->content; diff --git a/scripts/endpoints/author/1-fetch-single-author-mcpan-api.pl b/scripts/author/1-fetch-single-author.pl similarity index 72% rename from scripts/endpoints/author/1-fetch-single-author-mcpan-api.pl rename to scripts/author/1-fetch-single-author.pl index fe543ea..d2e5ea1 100755 --- a/scripts/endpoints/author/1-fetch-single-author-mcpan-api.pl +++ b/scripts/author/1-fetch-single-author.pl @@ -6,7 +6,7 @@ use Data::Printer; use MetaCPAN::Client; -my $mcpan = MetaCPAN::Client->new(); +my $mcpan = MetaCPAN::Client->new( version => 'v1' ); my $author = $mcpan->author('MSTROUT'); p $author; diff --git a/scripts/endpoints/author/1a-search-authors-mcpan-api.pl b/scripts/author/1a-search-authors.pl similarity index 85% rename from scripts/endpoints/author/1a-search-authors-mcpan-api.pl rename to scripts/author/1a-search-authors.pl index b2d76cb..2525ec6 100755 --- a/scripts/endpoints/author/1a-search-authors-mcpan-api.pl +++ b/scripts/author/1a-search-authors.pl @@ -7,7 +7,7 @@ use Data::Printer; use MetaCPAN::Client; -my $mc = MetaCPAN::Client->new; +my $mc = MetaCPAN::Client->new( version => 'v1' ); my $search = $mc->author( { name => 'Olaf *' } ); diff --git a/scripts/endpoints/author/1b-scroll-all-authors-es.pl b/scripts/author/1b-scroll-all-authors-es.pl similarity index 96% rename from scripts/endpoints/author/1b-scroll-all-authors-es.pl rename to scripts/author/1b-scroll-all-authors-es.pl index 6b7bb57..b24b0a3 100755 --- a/scripts/endpoints/author/1b-scroll-all-authors-es.pl +++ b/scripts/author/1b-scroll-all-authors-es.pl @@ -4,12 +4,13 @@ use warnings; use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $scroller = es()->scroll_helper( search_type => 'scan', scroll => '5m', - index => 'v0', + index => 'cpan', type => 'author', size => 100, body => { diff --git a/scripts/endpoints/author/1c-scroll-all-authors-with-twitter-es.pl b/scripts/author/1c-scroll-all-authors-with-twitter-es.pl similarity index 81% rename from scripts/endpoints/author/1c-scroll-all-authors-with-twitter-es.pl rename to scripts/author/1c-scroll-all-authors-with-twitter-es.pl index cbbdd9c..1b1e0fe 100755 --- a/scripts/endpoints/author/1c-scroll-all-authors-with-twitter-es.pl +++ b/scripts/author/1c-scroll-all-authors-with-twitter-es.pl @@ -4,21 +4,21 @@ use warnings; use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $scroller = es()->scroll_helper( search_type => 'scan', scroll => '5m', - index => 'v0', + index => 'cpan', type => 'author', size => 100, body => { query => { filtered => { - query => { term => { 'author.profile.name' => 'twitter' } }, + filter => { match => { 'profile.name' => 'twitter' } }, }, - match_all => {} - } + }, }, ); diff --git a/scripts/endpoints/author/2-twitter-or-github-es.pl b/scripts/author/2-twitter-or-github-es.pl similarity index 77% rename from scripts/endpoints/author/2-twitter-or-github-es.pl rename to scripts/author/2-twitter-or-github-es.pl index 157f5cb..32b8837 100755 --- a/scripts/endpoints/author/2-twitter-or-github-es.pl +++ b/scripts/author/2-twitter-or-github-es.pl @@ -4,37 +4,36 @@ use warnings; use feature qw( say ); +use lib './lib'; use MetaCPAN::Util qw( es ); my $scroller = es()->scroll_helper( search_type => 'scan', scroll => '5m', - index => 'v0', + index => 'cpan', type => 'author', size => 100, body => { query => { - match_all => {}, - filtered => { + filtered => { filter => { or => [ { and => [ { term => { - 'author.profile.name' => 'twitter' + 'profile.name' => 'twitter' } }, - { term => { 'author.country' => 'US' } } + { term => { 'country' => 'US' } } ] }, { and => [ { - term => - { 'author.profile.name' => 'github' } + term => { 'profile.name' => 'github' } }, - { term => { 'author.country' => 'CA' } } + { term => { 'country' => 'CA' } } ] }, ], diff --git a/scripts/author/twitter.pl b/scripts/author/twitter.pl new file mode 100755 index 0000000..c6a6508 --- /dev/null +++ b/scripts/author/twitter.pl @@ -0,0 +1,46 @@ +#!/usr/bin/env perl + +use strict; +use warnings; +use feature qw( say ); + +use MetaCPAN::Client; + +my $mc = MetaCPAN::Client->new( version => 'v1' ); + +# Search for authors with a listed Twitter account +my $search = $mc->author( { 'profile.name' => 'twitter' } ); + +my @handles; +while ( my $author = $search->next ) { + + # grep matching author profiles to extract only the Twitter id + my @profiles = grep { $_->{name} eq 'twitter' } @{ $author->profile }; + + foreach my $profile (@profiles) { + my $id = $profile->{id}; + + # not every handle returned by the API is prefixed by "@", so + # we'll add it when it's missing + $id = '@' . $id unless $id =~ m{\A@}; + push @handles, $id; + } +} + +# print a case-insensitive alpha-sorted list for humans to enjoy +say $_ for sort { "\L$a" cmp "\L$b" } @handles; + +=pod + +=head1 SYNOPSIS + +To create your own Twitter list of CPAN authors install and configure +L + +Then run: + + t list create cpan-authors + perl scripts/author/twitter.pl | xargs t list add cpan-authors + t list cpan-authors members + +=cut diff --git a/scripts/by-topic/1-fetch-author-curl.sh b/scripts/by-topic/1-fetch-author-curl.sh deleted file mode 120000 index c3c7c68..0000000 --- a/scripts/by-topic/1-fetch-author-curl.sh +++ /dev/null @@ -1 +0,0 @@ -../endpoints/author/1-fetch-single-author-curl.sh \ No newline at end of file diff --git a/scripts/by-topic/2-fetch-author-mcpan-api.pl b/scripts/by-topic/2-fetch-author-mcpan-api.pl deleted file mode 120000 index 5f2eecc..0000000 --- a/scripts/by-topic/2-fetch-author-mcpan-api.pl +++ /dev/null @@ -1 +0,0 @@ -../endpoints/author/1-fetch-single-author-mcpan-api.pl \ No newline at end of file diff --git a/scripts/by-topic/3-fetch-author-es.pl b/scripts/by-topic/3-fetch-author-es.pl deleted file mode 120000 index 3d3ca4f..0000000 --- a/scripts/by-topic/3-fetch-author-es.pl +++ /dev/null @@ -1 +0,0 @@ -../endpoints/author/1-fetch-single-author-es.pl \ No newline at end of file diff --git a/scripts/by-topic/4-scrolling-es.pl b/scripts/by-topic/4-scrolling-es.pl deleted file mode 120000 index 85542d8..0000000 --- a/scripts/by-topic/4-scrolling-es.pl +++ /dev/null @@ -1 +0,0 @@ -../endpoints/author/1b-scroll-all-authors-es.pl \ No newline at end of file diff --git a/scripts/by-topic/5-autocomplete-www-mech.pl b/scripts/by-topic/5-autocomplete-www-mech.pl deleted file mode 120000 index 40b1914..0000000 --- a/scripts/by-topic/5-autocomplete-www-mech.pl +++ /dev/null @@ -1 +0,0 @@ -../endpoints/search/autocomplete/www-mech.pl \ No newline at end of file diff --git a/scripts/by-topic/6-a-module-is-a-file-es.pl b/scripts/by-topic/6-a-module-is-a-file-es.pl deleted file mode 120000 index 92d034e..0000000 --- a/scripts/by-topic/6-a-module-is-a-file-es.pl +++ /dev/null @@ -1 +0,0 @@ -../endpoints/favorite/5-plus-plus-your-favorites-es.pl \ No newline at end of file diff --git a/scripts/endpoints/distribution/1-bugs-for-dist-es.pl b/scripts/distribution/1-bugs-for-dist-es.pl similarity index 83% rename from scripts/endpoints/distribution/1-bugs-for-dist-es.pl rename to scripts/distribution/1-bugs-for-dist-es.pl index 39c0e88..631ad5b 100755 --- a/scripts/endpoints/distribution/1-bugs-for-dist-es.pl +++ b/scripts/distribution/1-bugs-for-dist-es.pl @@ -4,10 +4,11 @@ use warnings; use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $dist = es()->get( - index => 'v0', + index => 'cpan', type => 'distribution', id => 'Moose', ); diff --git a/scripts/distribution/2-dists-with-rt-source.pl b/scripts/distribution/2-dists-with-rt-source.pl new file mode 100755 index 0000000..d591ac1 --- /dev/null +++ b/scripts/distribution/2-dists-with-rt-source.pl @@ -0,0 +1,16 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +use Data::Printer; +use MetaCPAN::Client; + +my $mc = MetaCPAN::Client->new( version => 'v1' ); + +my $search = $mc->all( 'distributions', + { es_filter => { exists => { field => 'bugs.rt.source' } } } ); + +while ( my $dist = $search->next ) { + p $dist; +} diff --git a/scripts/endpoints/author/1-fetch-single-author-curl.sh b/scripts/endpoints/author/1-fetch-single-author-curl.sh deleted file mode 100755 index 5822761..0000000 --- a/scripts/endpoints/author/1-fetch-single-author-curl.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/sh - -curl http://api.metacpan.org/v0/author/MSTROUT diff --git a/scripts/endpoints/author/1-fetch-single-author-es.pl b/scripts/endpoints/author/1-fetch-single-author-es.pl deleted file mode 100755 index fa3f2b9..0000000 --- a/scripts/endpoints/author/1-fetch-single-author-es.pl +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use Data::Printer; -use Search::Elasticsearch; - -my $es = Search::Elasticsearch->new( - cxn_pool => 'Static::NoPing', - nodes => 'api.metacpan.org', - trace_to => 'Stdout', -); - -my $author = $es->get( - index => 'v0', - type => 'author', - id => 'MSTROUT', -); - -p $author; diff --git a/scripts/endpoints/author/1-fetch-single-author-mcpan-api-tiny.pl b/scripts/endpoints/author/1-fetch-single-author-mcpan-api-tiny.pl deleted file mode 100755 index b4866a1..0000000 --- a/scripts/endpoints/author/1-fetch-single-author-mcpan-api-tiny.pl +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use Data::Printer; -use MetaCPAN::API::Tiny; - -my $mcpan = MetaCPAN::API::Tiny->new(); -my $author = $mcpan->author('MSTROUT'); - -p $author; diff --git a/scripts/endpoints/favorite/3-leaderboard-es.pl b/scripts/endpoints/favorite/3-leaderboard-es.pl deleted file mode 100755 index 1811b2a..0000000 --- a/scripts/endpoints/favorite/3-leaderboard-es.pl +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use Data::Printer; -use DateTime; -use MetaCPAN::Util qw( es ); - -my $now = DateTime->now; -my $then = $now->clone->subtract( months => 1 ); - -my $faves = es()->search( - index => 'v0', - type => 'favorite', - body => { - query => { match_all => {} }, - facets => { - dist => { - terms => { field => 'favorite.distribution', size => 40 }, - }, - }, - }, - size => 0, -); - -my @dists = map { $_->{terms} } $faves->{facets}->{dist}; - -p @dists; - -=pod - -=DESCRIPTION - -Because of the way facets are calculated, you may find that the numbers you get -towards the end of the list are not 100% accurate. You can test this by -checking the results for position #10 with a size of 10 versus a size of 20. -So, if you need exact numbers for the leaderboard (and why wouldn't you?), you -should set a size greater than what you actually require. You can experiment -with the size a bit to see the size required for the first X results to remain -consistent. Just with some quick testing, it looks like a size of 40 is a good -number in order to get consistent numbers for the top 10 results. - -=cut diff --git a/scripts/endpoints/module/2-fetch-all-modules.pl b/scripts/endpoints/module/2-fetch-all-modules.pl deleted file mode 100755 index c44da81..0000000 --- a/scripts/endpoints/module/2-fetch-all-modules.pl +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; -use feature qw( state ); - -use Data::Printer filters => { -external => ['JSON'] }; -use MetaCPAN::Util qw( es ); - -# distributions which MetaCPAN has chosen not to include in its search index -# feel free to include these in your searches if you wish -my @ROGUE_DISTRIBUTIONS - = qw(kurila perl_debug perl-5.005_02+apache1.3.3+modperl pod2texi perlbench spodcxx); - -my $scroller = es()->scroll_helper( - index => 'v0', - type => 'file', - body => { - query => { match_all => {} }, - filter => { - and => [ - { - not => { - filter => { - or => [ - map { - { term => { 'file.distribution' => $_ } } - } @ROGUE_DISTRIBUTIONS - ] - } - } - }, - { term => { status => 'latest' } }, - { - or => [ - - # we are looking for files that have no authorized - # property (e.g. .pod files) and files that are - # authorized - { missing => { field => 'file.authorized' } }, - { term => { 'file.authorized' => \1 } }, - ] - }, - { - or => [ - { - and => [ - { exists => { field => 'file.module.name' } }, - { term => { 'file.module.indexed' => \1 } } - ] - }, - { - and => [ - { exists => { field => 'documentation' } }, - { term => { 'file.indexed' => \1 } } - ] - } - ] - } - ] - }, - - sort => [ { "date" => "desc" } ], - fields => [ 'documentation', 'module' ], - }, - size => 10, -); - -# note that this search returns the names both of files which are pure -# documentation and also and arrayref of modules. keep in mind that one file -# can contain many modules. so, depending on your use case, you may want to -# deal with the documentation scalar, the module arrayref or even both. - -while ( my $result = $scroller->next ) { - state $counter = 0; - $counter++; - p $result->{fields}; - last if $counter == 10; -} - diff --git a/scripts/endpoints/pod/1-fetch-single-pod-doc-as-html-www-mech-cached.pl b/scripts/endpoints/pod/1-fetch-single-pod-doc-as-html-www-mech-cached.pl deleted file mode 100755 index 66c485d..0000000 --- a/scripts/endpoints/pod/1-fetch-single-pod-doc-as-html-www-mech-cached.pl +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; -use feature qw( say ); - -use WWW::Mechanize::Cached::GZip; -my $mech = WWW::Mechanize::Cached::GZip->new; - -$mech->get('http://api.metacpan.org/v0/pod/Carton'); - -say $mech->content; diff --git a/scripts/endpoints/pod/1a-fetch-single-pod-doc-as-plain-text-www-mech.pl b/scripts/endpoints/pod/1a-fetch-single-pod-doc-as-plain-text-www-mech.pl deleted file mode 100755 index 659ba5b..0000000 --- a/scripts/endpoints/pod/1a-fetch-single-pod-doc-as-plain-text-www-mech.pl +++ /dev/null @@ -1,11 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; -use feature qw( say ); - -use HTTP::Tiny; - -my $http = HTTP::Tiny->new( - default_headers => { 'Content-Type' => 'text/plain' } ); -say $http->get('http://api.metacpan.org/v0/pod/Carton')->{content}; diff --git a/scripts/endpoints/release/1-pkg2url-es.pl b/scripts/endpoints/release/1-pkg2url-es.pl deleted file mode 100755 index 836cb66..0000000 --- a/scripts/endpoints/release/1-pkg2url-es.pl +++ /dev/null @@ -1,25 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; -use feature qw( say ); - -use Search::Elasticsearch; - -my $es = Search::Elasticsearch->new( - cxn_pool => 'Static::NoPing', - nodes => 'api.metacpan.org', - trace_to => 'Stdout', -); - -my $release = $es->search( - index => 'v0', - type => 'release', - body => { - query => { match_all => {} }, - filter => - { term => { 'release.archive' => 'Acme-Hoge-0.03.tar.gz' } }, - }, -); - -say $release->{hits}{hits}[0]{_source}{download_url}; diff --git a/scripts/endpoints/release/1a-module2url-es.pl b/scripts/endpoints/release/1a-module2url-es.pl deleted file mode 100755 index 5e82240..0000000 --- a/scripts/endpoints/release/1a-module2url-es.pl +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; -use feature qw( say ); - -use Search::Elasticsearch; - -my $es = Search::Elasticsearch->new( - cxn_pool => 'Static::NoPing', - nodes => 'api.metacpan.org', - trace_to => 'Stdout', -); - -my $module = $es->search( - index => 'v0', - type => 'file', - body => { - query => { match_all => {} }, - filter => { - and => [ - { term => { 'file.authorized' => 'true' } }, - { term => { 'file.module.name' => 'Acme::Hoge' } }, - { term => { 'file.module.version' => '0.03' } } - ] - }, - }, -); - -my $release_name = $module->{hits}{hits}[0]{_source}{release}; - -my $release = $es->search( - index => 'v0', - type => 'release', - body => { - query => { match_all => {} }, - filter => { term => { 'release.name' => $release_name } }, - }, -); - -say $release->{hits}{hits}[0]{_source}{download_url}; diff --git a/scripts/endpoints/release/2-author-upload-leaderboard-es.pl b/scripts/endpoints/release/2-author-upload-leaderboard-es.pl deleted file mode 100755 index 2fbb6fe..0000000 --- a/scripts/endpoints/release/2-author-upload-leaderboard-es.pl +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use Data::Printer; -use DateTime; -use MetaCPAN::Util qw( es ); - -my $now = DateTime->now; -my $then = $now->clone->subtract( months => 1 ); - -my $faves = es()->search( - index => 'v0', - type => 'release', - body => { - query => { match_all => {} }, - facets => { - author => { terms => { field => 'release.author', size => 40 }, }, - }, - }, - size => 0, -); - -my @dists = map { $_->{terms} } $faves->{facets}->{dist}; - -p @dists; - -=pod - -=DESCRIPTION - -Because of the way facets are calculated, you may find that the numbers you get -towards the end of the list are not 100% accurate. You can test this by -checking the results for position #10 with a size of 10 versus a size of 20. -So, if you need exact numbers for the leaderboard (and why wouldn't you?), you -should set a size greater than what you actually require. You can experiment -with the size a bit to see the size required for the first X results to remain -consistent. - -=cut diff --git a/scripts/endpoints/release/3-author-uploads-one-author-es.pl b/scripts/endpoints/release/3-author-uploads-one-author-es.pl deleted file mode 100755 index 1871b2e..0000000 --- a/scripts/endpoints/release/3-author-uploads-one-author-es.pl +++ /dev/null @@ -1,27 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; -use feature qw( say ); - -use DateTime; -use MetaCPAN::Util qw( es ); - -my $uploads = es()->search( - index => 'v0', - type => 'release', - body => { - query => { - filtered => { - query => { match_all => {} }, - filter => { term => { 'release.author' => 'OALDERS' } }, - }, - }, - facets => { - author => { terms => { field => 'release.author', size => 40 }, }, - }, - }, - size => 0, -); - -say $uploads->{facets}->{author}->{terms}->[0]->{count}; diff --git a/scripts/endpoints/release/6-latest-releases-with-git-repo-es.pl b/scripts/endpoints/release/6-latest-releases-with-git-repo-es.pl deleted file mode 100755 index 868b39a..0000000 --- a/scripts/endpoints/release/6-latest-releases-with-git-repo-es.pl +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; -use feature qw( say ); - -use Data::Printer; -use MetaCPAN::Util qw( es ); - -my @must = ( - { term => { 'release.resources.repository.type' => 'git' }, }, - { term => { status => 'latest' } }, - { term => { authorized => 'true' } }, -); - -my $scroller = es()->scroll_helper( - body => { - query => { - filtered => { - query => { match_all => {} }, - filter => { bool => { must => \@must } }, - }, - }, - }, - fields => [ 'author', 'date', 'distribution', 'name', 'resources' ], - search_type => 'scan', - scroll => '5m', - index => 'v0', - type => 'release', - size => 500, -); - -while ( my $result = $scroller->next ) { - my $release = $result->{_source}; -} diff --git a/scripts/endpoints/search/reverse_dependencies/by-dist-www-mech.pl b/scripts/endpoints/search/reverse_dependencies/by-dist-www-mech.pl deleted file mode 100755 index 8e650ef..0000000 --- a/scripts/endpoints/search/reverse_dependencies/by-dist-www-mech.pl +++ /dev/null @@ -1,17 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use Data::Printer; -use JSON qw( decode_json ); -use WWW::Mechanize::GZip; -my $mech = WWW::Mechanize::GZip->new; - -$mech->get("http://api.metacpan.org/v0/search/reverse_dependencies/carton"); - -my $results = decode_json( $mech->content ); - -my @dists - = map { $_->{_source}->{distribution} } @{ $results->{hits}->{hits} }; -p @dists; diff --git a/scripts/endpoints/search/reverse_dependencies/by-release-version-mech.pl b/scripts/endpoints/search/reverse_dependencies/by-release-version-mech.pl deleted file mode 100755 index 3873047..0000000 --- a/scripts/endpoints/search/reverse_dependencies/by-release-version-mech.pl +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use Data::Printer; -use JSON qw( decode_json ); -use WWW::Mechanize; -my $mech = WWW::Mechanize->new; - -$mech->get( - "http://api.metacpan.org/v0/search/reverse_dependencies/MIYAGAWA/carton-v0.9.13" -); - -my $results = decode_json( $mech->content ); - -my @dists - = map { $_->{_source}->{distribution} } @{ $results->{hits}->{hits} }; -p @dists; diff --git a/scripts/endpoints/favorite/1-last-50-favorited-dists-es.pl b/scripts/favorite/1-last-50-favorited-dists-es.pl similarity index 90% rename from scripts/endpoints/favorite/1-last-50-favorited-dists-es.pl rename to scripts/favorite/1-last-50-favorited-dists-es.pl index 1c266a1..5901d91 100755 --- a/scripts/endpoints/favorite/1-last-50-favorited-dists-es.pl +++ b/scripts/favorite/1-last-50-favorited-dists-es.pl @@ -4,10 +4,11 @@ use warnings; use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $faves = es()->search( - index => 'v0', + index => 'cpan', type => 'favorite', body => { query => { match_all => {} }, diff --git a/scripts/endpoints/favorite/1a-last-100-favorited-dists-by-user-es.pl b/scripts/favorite/1a-last-100-favorited-dists-by-user-es.pl similarity index 83% rename from scripts/endpoints/favorite/1a-last-100-favorited-dists-by-user-es.pl rename to scripts/favorite/1a-last-100-favorited-dists-by-user-es.pl index acbb712..41ed3cc 100755 --- a/scripts/endpoints/favorite/1a-last-100-favorited-dists-by-user-es.pl +++ b/scripts/favorite/1a-last-100-favorited-dists-by-user-es.pl @@ -4,6 +4,7 @@ use warnings; use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $id = shift @ARGV; @@ -11,13 +12,13 @@ die "usage: ./bin/carton $0 \$user_id" if !$id; my $faves = es()->search( - index => 'v0', + index => 'cpan', type => 'favorite', body => { query => { filtered => { query => { match_all => {} }, - filter => { term => { 'favorite.user' => $id } } + filter => { term => { 'user' => $id } } }, }, sort => [ { date => 'desc' } ], diff --git a/scripts/endpoints/favorite/2-favorites-previous-month-es.pl b/scripts/favorite/2-favorites-previous-month-es.pl similarity index 84% rename from scripts/endpoints/favorite/2-favorites-previous-month-es.pl rename to scripts/favorite/2-favorites-previous-month-es.pl index 7335310..252760f 100755 --- a/scripts/endpoints/favorite/2-favorites-previous-month-es.pl +++ b/scripts/favorite/2-favorites-previous-month-es.pl @@ -5,21 +5,21 @@ use Data::Printer; use DateTime; +use lib './lib'; use MetaCPAN::Util qw( es ); my $now = DateTime->now; my $then = $now->clone->subtract( months => 1 ); my $faves = es()->search( - index => 'v0', + index => 'cpan', type => 'favorite', body => { query => { filtered => { - query => { match_all => {} }, filter => { range => { - 'favorite.date' => + date => { from => $then->datetime, to => $now->datetime } }, }, diff --git a/scripts/favorite/3-leaderboard-es.pl b/scripts/favorite/3-leaderboard-es.pl new file mode 100755 index 0000000..2b68b35 --- /dev/null +++ b/scripts/favorite/3-leaderboard-es.pl @@ -0,0 +1,34 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +use Data::Printer; +use lib './lib'; +use MetaCPAN::Util qw( es ); + +my $faves = es()->search( + index => 'cpan', + type => 'favorite', + body => { + aggs => { + dist => { + terms => { field => 'distribution', size => 10 }, + }, + }, + }, +); + +my @counts = map { +{ $_->{key} => $_->{doc_count} } } + @{ $faves->{aggregations}->{dist}->{buckets} }; +p @counts; + +__END__ +=pod + +=head1 DESCRIPTION + +Get the 10 distributions with the most ++ clicks, sorted by descending +popularity. + +=cut diff --git a/scripts/endpoints/favorite/4-leaderboard-previous-month-es.pl b/scripts/favorite/4-leaderboard-previous-month-es.pl similarity index 66% rename from scripts/endpoints/favorite/4-leaderboard-previous-month-es.pl rename to scripts/favorite/4-leaderboard-previous-month-es.pl index 782b1f7..f8e1cc5 100755 --- a/scripts/endpoints/favorite/4-leaderboard-previous-month-es.pl +++ b/scripts/favorite/4-leaderboard-previous-month-es.pl @@ -5,35 +5,35 @@ use Data::Printer; use DateTime; +use lib './lib'; use MetaCPAN::Util qw( es ); -my $now = DateTime->now; +my $now = DateTime->now; my $then = $now->clone->subtract( months => 1 ); my $faves = es()->search( - index => 'v0', + index => 'cpan', type => 'favorite', body => { query => { filtered => { - query => { match_all => {} }, filter => { range => { - 'favorite.date' => + date => { from => $then->datetime, to => $now->datetime } }, }, }, }, - facets => { + aggs => { dist => { - terms => { field => 'favorite.distribution', size => 50 }, + terms => { field => 'distribution', size => 50 }, }, }, }, size => 0, ); -my @dists = map { $_->{terms} } $faves->{facets}->{dist}; +my @dists = @{ $faves->{aggregations}{dist}{buckets} }; p @dists; diff --git a/scripts/endpoints/favorite/5-plus-plus-your-favorites-es.pl b/scripts/favorite/5-plus-plus-your-favorites-es.pl similarity index 73% rename from scripts/endpoints/favorite/5-plus-plus-your-favorites-es.pl rename to scripts/favorite/5-plus-plus-your-favorites-es.pl index 4898361..1ae1e10 100755 --- a/scripts/endpoints/favorite/5-plus-plus-your-favorites-es.pl +++ b/scripts/favorite/5-plus-plus-your-favorites-es.pl @@ -4,8 +4,9 @@ use warnings; use Data::Printer; -use HTTP::Tiny; -use JSON; +use HTTP::Tiny (); +use JSON::MaybeXS qw( encode_json ); +use lib './lib'; use MetaCPAN::Util qw( es ); my $token = shift @ARGV; @@ -15,22 +16,21 @@ unless @modules; my $module = es()->search( - index => 'v0', + index => 'cpan', type => 'file', fields => 'release', size => scalar @modules, body => { query => { filtered => { - query => { match_all => {} }, filter => { bool => { # a module is a file must => [ - { term => { 'file.authorized' => 'true' } }, - { terms => { 'file.module.name' => \@modules } }, - { term => { 'file.status' => 'latest' } } + { term => { 'authorized' => 'true' } }, + { terms => { 'module.name' => \@modules } }, + { term => { 'status' => 'latest' } } ] }, }, @@ -43,14 +43,13 @@ = map { $_->{fields}->{release} } @{ $module->{hits}->{hits} }; my $release = es()->search( - index => 'v0', + index => 'cpan', type => 'release', size => scalar @release_names, body => { query => { filtered => { - query => { match_all => {} }, - filter => { terms => { 'release.name' => \@release_names } }, + filter => { terms => { 'name' => \@release_names } }, }, }, }, @@ -70,9 +69,9 @@ sub plus_plus { my $params = shift; my $ua = HTTP::Tiny->new; my $res = $ua->post( - "https://api.metacpan.org/user/favorite?access_token=$token", + "https://fastapi.metacpan.org/v1/user/favorite?access_token=$token", { - content => to_json($params), + content => encode_json($params), headers => { 'content-type' => 'application/json' } }, ); diff --git a/scripts/endpoints/favorite/6-list-plussers-by-module.pl b/scripts/favorite/6-list-plussers-by-module.pl similarity index 83% rename from scripts/endpoints/favorite/6-list-plussers-by-module.pl rename to scripts/favorite/6-list-plussers-by-module.pl index f0ba8ff..49a4127 100755 --- a/scripts/endpoints/favorite/6-list-plussers-by-module.pl +++ b/scripts/favorite/6-list-plussers-by-module.pl @@ -5,6 +5,7 @@ use feature qw( say ); use MetaCPAN::Client; +use lib './lib'; use MetaCPAN::Util qw( es ); binmode( STDOUT, ":utf8" ); @@ -16,16 +17,14 @@ my $module = MetaCPAN::Client->new->module($module_name); my $plussers = es()->search( - index => 'v0', + index => 'cpan', type => 'favorite', size => 1000, body => { query => { filtered => { - query => { match_all => {} }, filter => { - term => - { 'favorite.distribution' => $module->distribution } + term => { 'distribution' => $module->distribution } }, }, }, @@ -37,14 +36,14 @@ my $total = @ids; my $authors = es()->search( - index => 'v0', + index => 'cpan', type => 'author', size => scalar @ids, body => { query => { filtered => { query => { match_all => {} }, - filter => { terms => { 'author.user' => \@ids } }, + filter => { terms => { 'user' => \@ids } }, }, }, fields => [ 'pauseid', 'name' ], diff --git a/scripts/endpoints/file/1-get-files-in-dist-es.pl b/scripts/file/1-get-files-in-dist-es.pl similarity index 70% rename from scripts/endpoints/file/1-get-files-in-dist-es.pl rename to scripts/file/1-get-files-in-dist-es.pl index 0d8aa71..f80f37d 100755 --- a/scripts/endpoints/file/1-get-files-in-dist-es.pl +++ b/scripts/file/1-get-files-in-dist-es.pl @@ -5,10 +5,11 @@ use feature qw( say ); use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $files = es()->search( - index => 'v0', + index => 'cpan', type => 'file', size => 300, body => { @@ -18,11 +19,9 @@ filter => { bool => { must => { - term => - { 'file.release' => 'HTML-Restrict-2.1.5' }, + term => { 'release' => 'HTML-Restrict-2.1.5' }, }, - must_not => - { term => { 'file.directory' => 'true' }, }, + must_not => { term => { 'directory' => 'true' }, }, }, }, }, diff --git a/scripts/endpoints/file/2-get-dists-with-cpanfile.pl b/scripts/file/2-get-dists-with-cpanfile.pl similarity index 94% rename from scripts/endpoints/file/2-get-dists-with-cpanfile.pl rename to scripts/file/2-get-dists-with-cpanfile.pl index c9f7f3e..224837e 100755 --- a/scripts/endpoints/file/2-get-dists-with-cpanfile.pl +++ b/scripts/file/2-get-dists-with-cpanfile.pl @@ -5,10 +5,11 @@ use feature qw( say ); use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $files = es()->search( - index => 'v0', + index => 'cpan', type => 'file', size => 10, body => { diff --git a/scripts/endpoints/file/3-find-files-in-top-level-by-name.pl b/scripts/file/3-find-files-in-top-level-by-name.pl similarity index 89% rename from scripts/endpoints/file/3-find-files-in-top-level-by-name.pl rename to scripts/file/3-find-files-in-top-level-by-name.pl index c9f7f3e..aeafec4 100755 --- a/scripts/endpoints/file/3-find-files-in-top-level-by-name.pl +++ b/scripts/file/3-find-files-in-top-level-by-name.pl @@ -5,16 +5,16 @@ use feature qw( say ); use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $files = es()->search( - index => 'v0', + index => 'cpan', type => 'file', size => 10, body => { query => { filtered => { - query => { match_all => {} }, filter => { and => [ { term => { 'path' => 'cpanfile' } }, diff --git a/scripts/endpoints/file/4-main-search.pl b/scripts/file/4-main-search.pl old mode 100644 new mode 100755 similarity index 89% rename from scripts/endpoints/file/4-main-search.pl rename to scripts/file/4-main-search.pl index 337dcad..40f8c0d --- a/scripts/endpoints/file/4-main-search.pl +++ b/scripts/file/4-main-search.pl @@ -4,7 +4,8 @@ use warnings; use Data::Printer; -use JSON qw( decode_json ); +use JSON::MaybeXS qw( decode_json ); +use lib './lib'; use MetaCPAN::Util qw( es ); my $search_term = shift @ARGV || 'HTML-Re'; @@ -13,7 +14,7 @@ } my $result = es()->search( - index => 'v0', + index => 'cpan', type => 'file', body => { query => { @@ -24,7 +25,7 @@ should => [ { term => { - "file.documentation" => { + "documentation" => { boost => 20, value => $search_term, } @@ -32,7 +33,7 @@ }, { term => { - "file.module.name" => { + "module.name" => { boost => 20, value => $search_term, }, @@ -48,10 +49,10 @@ default_operator => "AND", fields => [ "documentation.analyzed^2", - "file.module.name.analyzed^2", + "module.name.analyzed^2", "distribution.analyzed", "documentation.camelcase", - "file.module.name.camelcase", + "module.name.camelcase", "distribution.camelcase", ], query => $search_term, @@ -78,7 +79,7 @@ }, negative => { term => { - "file.mime" => { value => "text/x-script.perl" } + "mime" => { value => "text/x-script.perl" } } } } diff --git a/scripts/file/5-size-of-cpan.pl b/scripts/file/5-size-of-cpan.pl new file mode 100755 index 0000000..5689597 --- /dev/null +++ b/scripts/file/5-size-of-cpan.pl @@ -0,0 +1,26 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +use Data::Printer; +use MetaCPAN::Client (); + +my $mc = MetaCPAN::Client->new( version => 'v1' ); + +my $file = $mc->all( + 'files', + { + aggregations => { aggs => { sum => { field => 'stat.size' } } }, + } +); +p $file->aggregations; + +__END__ +=pod + +=head1 DESCRIPTION + +Get the size of CPAN + BackPAN, when it's unpacked. + +=cut diff --git a/scripts/endpoints/module/1-fetch-single-module-es.pl b/scripts/module/1-fetch-single-module-es.pl similarity index 83% rename from scripts/endpoints/module/1-fetch-single-module-es.pl rename to scripts/module/1-fetch-single-module-es.pl index 46bd1fe..6698c11 100755 --- a/scripts/endpoints/module/1-fetch-single-module-es.pl +++ b/scripts/module/1-fetch-single-module-es.pl @@ -4,10 +4,11 @@ use warnings; use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $module = es()->get( - index => 'v0', + index => 'cpan', type => 'module', id => 'HTML::Restrict', ); diff --git a/scripts/module/3-fetch-modules-in-release.pl b/scripts/module/3-fetch-modules-in-release.pl new file mode 100755 index 0000000..95438c8 --- /dev/null +++ b/scripts/module/3-fetch-modules-in-release.pl @@ -0,0 +1,30 @@ +use strict; +use warnings; +use feature qw( say ); + +use MetaCPAN::Client (); +my $client = MetaCPAN::Client->new( version => 'v1' ); + +my $dist = shift @ARGV; + +die "usage: ./bin/carton $0 Moose" unless $dist; + +my $modules = $client->module( + { + all => [ + { authorized => 'true' }, + { binary => 'false' }, + { distribution => $dist }, + { indexed => 'true' }, + { status => 'latest' }, + ] + } +); + +while ( my $module = $modules->next ) { + next unless $module->module; + for my $pkg ( @{ $module->module } ) { + say $pkg->{name}; + } +} + diff --git a/scripts/permission/aggregate-author-perms.pl b/scripts/permission/aggregate-author-perms.pl new file mode 100755 index 0000000..e8decbb --- /dev/null +++ b/scripts/permission/aggregate-author-perms.pl @@ -0,0 +1,178 @@ +#!/usr/bin/env perl + +use strict; +use warnings; +use feature qw( say ); + +use CLDR::Number (); +use Cpanel::JSON::XS qw( decode_json ); +use Data::Printer; +use List::Compare (); +use LWP::UserAgent (); +use Math::Round qw( nearest ); +use Parse::CPAN::Packages::Fast (); +use WWW::Mechanize::Cached (); + +my $cldr = CLDR::Number->new( locale => 'en' ); +my $decf = $cldr->decimal_formatter; +my $perf = $cldr->percent_formatter( minimum_fraction_digits => 2 ); + +my @maxmind_authors = ( + 'OALDERS', 'EILARA', 'TJMATHER', 'MATEU', 'OSCHWALD', 'RSRCHBOY', + 'FLORA', 'MARKF', 'RUBEN', 'WDS', 'KLPTWO', 'PCRONIN', + 'ANDYJACK', 'MAXMIND', +); +my @maxmind_redacted_authors = ( + 'OALDERS', 'EILARA', 'TJMATHER', 'MATEU', 'OSCHWALD', 'RSRCHBOY', + 'MARKF', 'RUBEN', 'WDS', 'KLPTWO', 'PCRONIN', 'ANDYJACK', +); + +my %PTS = ( + 'qa2010' => [ + 'CHORNY', 'HMBRAND', 'RURBAN', 'MARCEL', + 'BOOK', 'BDFOY', 'ANDYA', 'POTYL', + 'DAXIM', 'ABELTJE', 'OVID', 'HORNBURG', + 'DOMM', 'MIYAGAWA', 'PJCJ', 'BARBIE', + 'FLORA', 'RJBS', 'JKUTEJ', 'SREZIC', + 'MSCHWERN', 'SZABGAB', 'RGIERSIG', 'SQUEEK', + 'PEPL' + ], + 'qa2011' => [ + 'RGE', 'HMBRAND', 'MARCEL', 'SCHWIGON', + 'LEONT', 'BOOK', 'BDFOY', 'ANDYA', + 'AVAR', 'POTYL', 'DAXIM', 'ABELTJE', + 'SMUELLER', 'OVID', 'DOLMEN', 'WESJDJ', + 'SAPER', 'ADAMK', 'PJCJ', 'FLORA', + 'ABIGAIL', 'RJBS', 'PERLER', 'DGL', + 'JKUTEJ', 'WONKO', 'ADIE', 'BURAK' + ], + 'qa2012' => [ + 'XAV', 'HMBRAND', 'CHESSKIT', 'NPEREZ', + 'RGARCIA', 'ELIZABETH', 'DAGOLDEN', 'SCHWIGON', + 'LEONT', 'BOOK', 'BDFOY', 'ANDYA', + 'HAGGAI', 'GETTY', 'DAXIM', 'DDUMONT', + 'ABELTJE', 'OVID', 'ELBEHO', 'WOLFSAGE', + 'DOLMEN', 'WESJDJ', 'PAUAMMA', 'SAPER', + 'OALDERS', 'MIYAGAWA', 'PJCJ', 'BARBIE', + 'FLORA', 'VPIT', 'RJBS', 'DGL', + 'ISHIGAKI', 'WONKO', 'SREZIC', 'MSCHWERN', + 'GARU', 'RIBASUSHI', 'ADIE', 'APEIRON', + 'ANDK' + ], + 'qa2013' => [ + 'PERRETTDL', 'SJN', 'TBSLIVER', 'JROBINSON', + 'MITHALDU', 'HMBRAND', 'ELIZABETH', 'DAGOLDEN', + 'SCHWIGON', 'JKEENAN', 'LEONT', 'BOOK', + 'BANNAN', 'ANDYA', 'DREBOLO', 'PDCAWLEY', + 'GETTY', 'BINGOS', 'ABELTJE', 'REHSACK', + 'BYTEROCK', 'WOLFSAGE', 'DOLMEN', 'BBUSS', + 'PJCJ', 'BARBIE', 'ARC', 'RJBS', + 'DGL', 'ISHIGAKI', 'RIBASUSHI', 'JMASTROS', + 'ANDK', 'NEWELLC' + ], + 'qa2014' => [ + 'SJN', 'ETHER', 'MITHALDU', 'HMBRAND', + 'ELIZABETH', 'DAGOLDEN', 'SCHWIGON', 'LEONT', + 'BOOK', 'DAMS', 'MSTROUT', 'ABELTJE', + 'TIMB', 'FROGGS', 'OVID', 'REHSACK', + 'ELBEHO', 'WOLFSAGE', 'DOLMEN', 'SAPER', + 'OALDERS', 'PJCJ', 'BARBIE', 'RJBS', + 'ISHIGAKI', 'NEILB', 'SREZIC', 'RIBASUSHI', + 'HAARG', 'ANDK' + ], + 'qa2015' => [ + 'SJN', 'ETHER', 'MITHALDU', 'HMBRAND', + 'ELIZABETH', 'DAGOLDEN', 'SCHWIGON', 'LEONT', + 'EXODIST', 'BOOK', 'TADZIK', 'TINITA', + 'ABELTJE', 'INGY', 'FROGGS', 'REHSACK', + 'WOLFSAGE', 'DOLMEN', 'OALDERS', 'WOLLMERS', + 'MIYAGAWA', 'PJCJ', 'BARBIE', 'BARTOLIN', + 'RJBS', 'ISHIGAKI', 'NEILB', 'SREZIC', + 'DRTECH', 'RIBASUSHI', 'LICHTKIND', 'ARISTOTLE', + 'ANDK', 'NINE' + ], + 'qa2016' => [ + 'ETHER', 'HMBRAND', 'LLAP', 'ELIZABETH', + 'SCHWIGON', 'JKEENAN', 'LEONT', 'EXODIST', + 'BOOK', 'TADZIK', 'MICKEY', 'BINGOS', + 'ABELTJE', 'TIMB', 'JBERGER', 'WOLFSAGE', + 'DOLMEN', 'OALDERS', 'PJCJ', 'BARBIE', + 'ARC', 'RJBS', 'ISHIGAKI', 'XSAWYERX', + 'NEILB', 'SREZIC', 'ARISTOTLE', 'SARGIE', + 'ANDK' + ], + 'qa2017' => [ + 'ETHER', 'MITHALDU', 'HMBRAND', 'LEEJO', + 'LLAP', 'SKAJI', 'ELIZABETH', 'LEONT', + 'UGEXE', 'BOOK', 'TODDR', 'PREACTION', + 'TADZIK', 'MICKEY', 'BINGOS', 'TINITA', + 'ABELTJE', 'INGY', 'JBERGER', 'ELBEHO', + 'WOLFSAGE', 'OALDERS', 'MIYAGAWA', 'PJCJ', + 'ARC', 'ETJ', 'ISHIGAKI', 'XSAWYERX', + 'NEILB', 'SREZIC', 'GARU', 'ATOOMIC', + 'ARISTOTLE', 'HAARG', 'ANDK', 'NINE' + ], +); + +my $ua = LWP::UserAgent->new; +$ua->mirror( 'https://cpan.metacpan.org/modules/02packages.details.txt', + '02packages.details.txt' ); + +my $parser = Parse::CPAN::Packages::Fast->new('02packages.details.txt'); + +say join '|', + ( + q{}, + q{}, + 'Modules with maint', + 'Modules in 02packages', + '% of modules in 02packages', + ); +say join '---', ( ('|') x 5 ); + +for my $author ( sort @{$PTS{qa2017}} ) { + crunch_numbers( $author, [$author]); +} + +for my $group_name ( sort keys %PTS ) { + crunch_numbers( $group_name, $PTS{$group_name} ); +} + +sub crunch_numbers { + my $title = shift; + my $authors = shift; + + # The modules which these authors have release permissions on. + my %perms = get_permissions($authors); + + my $lc = List::Compare->new( [ $parser->packages ], [ keys %perms ] ); + + # The permissioned modules which actually appear in 02packages. + my @covered = $lc->get_intersection; + + my $percent + = nearest( 0.0001, scalar @covered / ( scalar $parser->packages ) ); + + say join '|', $title, $decf->format( scalar keys %perms ), + $decf->format( scalar @covered ), $perf->format($percent); +} + +sub get_permissions { + my $authors = shift; + my $mech = WWW::Mechanize::Cached->new; + + my %modules; + + my $base_url = 'http://fastapi.metacpan.org/v1/permission/by_author'; + foreach my $author ( @{$authors} ) { + my $url = "$base_url/$author"; + $mech->get($url); + my $perms = decode_json( $mech->content ); + + foreach my $perm ( @{ $perms->{permissions} } ) { + push @{ $modules{ $perm->{module_name} } }, $author; + } + } + + return %modules; +} diff --git a/scripts/endpoints/pod/1-fetch-single-pod-doc-as-html-mcpan-api.pl b/scripts/pod/1-fetch-single-pod-doc-as-html.pl similarity index 72% rename from scripts/endpoints/pod/1-fetch-single-pod-doc-as-html-mcpan-api.pl rename to scripts/pod/1-fetch-single-pod-doc-as-html.pl index 3de9771..4ea6c9c 100755 --- a/scripts/endpoints/pod/1-fetch-single-pod-doc-as-html-mcpan-api.pl +++ b/scripts/pod/1-fetch-single-pod-doc-as-html.pl @@ -6,6 +6,6 @@ use MetaCPAN::Client; -my $mcpan = MetaCPAN::Client->new; +my $mcpan = MetaCPAN::Client->new( version => 'v1' ); my $html_pod = $mcpan->pod('Carton'); say $html_pod->html; diff --git a/scripts/endpoints/pod/1a-fetch-single-pod-doc-as-plain-text-mcpan-api.pl b/scripts/pod/1a-fetch-single-pod-doc-as-plain-text.pl similarity index 80% rename from scripts/endpoints/pod/1a-fetch-single-pod-doc-as-plain-text-mcpan-api.pl rename to scripts/pod/1a-fetch-single-pod-doc-as-plain-text.pl index bbfe287..5bdd370 100755 --- a/scripts/endpoints/pod/1a-fetch-single-pod-doc-as-plain-text-mcpan-api.pl +++ b/scripts/pod/1a-fetch-single-pod-doc-as-plain-text.pl @@ -4,7 +4,7 @@ use warnings; use feature qw( say ); -use MetaCPAN::Client; +use MetaCPAN::Client( version => 'v1' ); my $mcpan = MetaCPAN::Client->new; my $html_pod = $mcpan->pod('Carton'); diff --git a/scripts/endpoints/pod/README b/scripts/pod/README old mode 100644 new mode 100755 similarity index 100% rename from scripts/endpoints/pod/README rename to scripts/pod/README diff --git a/scripts/release/1-pkg2url-es.pl b/scripts/release/1-pkg2url-es.pl new file mode 100755 index 0000000..9d5f94a --- /dev/null +++ b/scripts/release/1-pkg2url-es.pl @@ -0,0 +1,18 @@ +#!/usr/bin/env perl + +use strict; +use warnings; +use feature qw( say ); + +use lib './lib'; +use MetaCPAN::Util qw( es ); + +my $release = es->search( + index => 'cpan', + type => 'release', + body => { + filter => { term => { 'archive' => 'Acme-Hoge-0.03.tar.gz' } }, + }, +); + +say $release->{hits}{hits}[0]{_source}{download_url}; diff --git a/scripts/release/10-all-latest-releases.pl b/scripts/release/10-all-latest-releases.pl new file mode 100755 index 0000000..db8334b --- /dev/null +++ b/scripts/release/10-all-latest-releases.pl @@ -0,0 +1,12 @@ +use strict; +use warnings; +use feature qw( say ); + +use MetaCPAN::Client (); + +my $mc = MetaCPAN::Client->new; +my $release_results = $mc->release( { status => 'latest' } ); + +while ( my $release = $release_results->next ) { + say $release->download_url; +} diff --git a/scripts/release/11-all-latest-releases-by-NEILB.pl b/scripts/release/11-all-latest-releases-by-NEILB.pl new file mode 100755 index 0000000..851663d --- /dev/null +++ b/scripts/release/11-all-latest-releases-by-NEILB.pl @@ -0,0 +1,14 @@ +use strict; +use warnings; +use feature qw( say ); + +use MetaCPAN::Client (); + +my $mc = MetaCPAN::Client->new; +my $release_results + = $mc->release( + { all => [ { author => 'NEILB', }, { status => 'latest' } ] } ); + +while ( my $release = $release_results->next ) { + say $release->download_url; +} diff --git a/scripts/release/12-all-latest-releases-by-NEILB-with-git-repository.pl b/scripts/release/12-all-latest-releases-by-NEILB-with-git-repository.pl new file mode 100755 index 0000000..79973f3 --- /dev/null +++ b/scripts/release/12-all-latest-releases-by-NEILB-with-git-repository.pl @@ -0,0 +1,20 @@ +use strict; +use warnings; +use feature qw( say ); + +use MetaCPAN::Client (); + +my $mc = MetaCPAN::Client->new; +my $release_results = $mc->release( + { + all => [ + { author => 'NEILB', }, + { status => 'latest' }, + { 'resources.repository.type' => 'git' } + ] + } +); + +while ( my $release = $release_results->next ) { + say $release->resources->{repository}->{url}; +} diff --git a/scripts/release/13-all-releases-in-last-24-hours.pl b/scripts/release/13-all-releases-in-last-24-hours.pl new file mode 100644 index 0000000..7792f11 --- /dev/null +++ b/scripts/release/13-all-releases-in-last-24-hours.pl @@ -0,0 +1,24 @@ +use strict; +use warnings; +use feature qw( say ); + +use DateTime (); +use MetaCPAN::Client (); + +my $now = DateTime->now; +my $then = DateTime->now->subtract( days => 1 ); + +my $mc = MetaCPAN::Client->new; + +my $release_set = $mc->all( + 'releases', + { + es_filter => { + range => { date => { from => $then->datetime } }, + }, + } +); + +while ( my $release = $release_set->next ) { + say $release->download_url; +} diff --git a/scripts/release/1a-module2url-es.pl b/scripts/release/1a-module2url-es.pl new file mode 100755 index 0000000..9ffd498 --- /dev/null +++ b/scripts/release/1a-module2url-es.pl @@ -0,0 +1,37 @@ +#!/usr/bin/env perl + +use strict; +use warnings; +use feature qw( say ); + +use Search::Elasticsearch; + +use lib './lib'; +use MetaCPAN::Util qw( es ); + +my $module = es->search( + index => 'cpan', + type => 'file', + body => { + query => { match_all => {} }, + filter => { + and => [ + { term => { 'authorized' => 'true' } }, + { term => { 'module.name' => 'Acme::Hoge' } }, + { term => { 'module.version' => '0.03' } } + ] + }, + }, +); + +my $release_name = $module->{hits}{hits}[0]{_source}{release}; + +my $release = es->search( + index => 'cpan', + type => 'release', + body => { + filter => { term => { 'name' => $release_name } }, + }, +); + +say $release->{hits}{hits}[0]{_source}{download_url}; diff --git a/scripts/release/2-author-upload-leaderboard-es.pl b/scripts/release/2-author-upload-leaderboard-es.pl new file mode 100755 index 0000000..c71bd65 --- /dev/null +++ b/scripts/release/2-author-upload-leaderboard-es.pl @@ -0,0 +1,25 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +use Data::Printer; + +use lib './lib'; +use MetaCPAN::Util qw( es ); + +my $uploads = es()->search( + index => 'cpan', + type => 'release', + body => { + query => { match_all => {} }, + aggs => { + author => { terms => { field => 'author', size => 10 }, }, + }, + }, +); + +my @authors = map { +{ $_->{key} => $_->{doc_count} } } + @{ $uploads->{aggregations}->{author}->{buckets} }; + +p @authors; diff --git a/scripts/release/3-author-uploads-one-author-es.pl b/scripts/release/3-author-uploads-one-author-es.pl new file mode 100755 index 0000000..062a860 --- /dev/null +++ b/scripts/release/3-author-uploads-one-author-es.pl @@ -0,0 +1,27 @@ +#!/usr/bin/env perl + +use strict; +use warnings; +use feature qw( say ); + +use lib './lib'; +use MetaCPAN::Util qw( es ); + +my $uploads = es()->search( + index => 'cpan', + type => 'release', + body => { + query => { + filtered => { + filter => { term => { 'author' => 'OALDERS' } }, + }, + }, + aggs => { + author => { terms => { field => 'author', size => 40 }, }, + }, + }, + size => 0, +); + +use DDP; +p $uploads; diff --git a/scripts/endpoints/release/4-latest-release-versions-es.pl b/scripts/release/4-latest-release-versions-es.pl similarity index 91% rename from scripts/endpoints/release/4-latest-release-versions-es.pl rename to scripts/release/4-latest-release-versions-es.pl index e35e144..4ac4f55 100755 --- a/scripts/endpoints/release/4-latest-release-versions-es.pl +++ b/scripts/release/4-latest-release-versions-es.pl @@ -4,10 +4,12 @@ use warnings; use Data::Printer; + +use lib './lib'; use MetaCPAN::Util qw( es ); my $latest = es()->search( - index => 'v0', + index => 'cpan', type => 'release', fields => [ 'distribution', 'version' ], size => 3, @@ -17,10 +19,10 @@ query => { match_all => {} }, filter => { and => [ - { term => { 'release.status' => 'latest' } }, + { term => { 'status' => 'latest' } }, { terms => { - 'release.distribution' => + 'distribution' => [ 'Moose', 'MetaCPAN-API', 'DBIx-Class' ] }, }, diff --git a/scripts/endpoints/release/4a-latest-release-versions-bool-filter-es.pl b/scripts/release/4a-latest-release-versions-bool-filter-es.pl similarity index 71% rename from scripts/endpoints/release/4a-latest-release-versions-bool-filter-es.pl rename to scripts/release/4a-latest-release-versions-bool-filter-es.pl index 8a89aaa..fbfcc2f 100755 --- a/scripts/endpoints/release/4a-latest-release-versions-bool-filter-es.pl +++ b/scripts/release/4a-latest-release-versions-bool-filter-es.pl @@ -4,32 +4,31 @@ use warnings; use Data::Printer; +use lib './lib'; use MetaCPAN::Util qw( es ); my $latest = es()->search( - index => 'v0', + index => 'cpan', type => 'release', fields => [ 'distribution', 'version' ], - size => 3, + size => 4, body => { query => { filtered => { - query => { match_all => {} }, filter => { bool => { must => [ - { term => { 'release.status' => 'latest' } }, + { term => { 'status' => 'latest' } }, { terms => { - 'release.distribution' => [ - 'Moose', 'MetaCPAN-API', - 'DBIx-Class' + 'distribution' => [ + 'Moose', 'MetaCPAN-Client', + 'DBIx-Class', 'Moo', ] }, }, ], - must_not => - { term => { 'release.author' => 'ETHER' } }, + must_not => { term => { 'author' => 'ETHER' } }, }, }, }, diff --git a/scripts/endpoints/release/5-latest-releases-by-author-es.pl b/scripts/release/5-latest-releases-by-author-es.pl similarity index 85% rename from scripts/endpoints/release/5-latest-releases-by-author-es.pl rename to scripts/release/5-latest-releases-by-author-es.pl index 82181b0..9f6dbff 100755 --- a/scripts/endpoints/release/5-latest-releases-by-author-es.pl +++ b/scripts/release/5-latest-releases-by-author-es.pl @@ -4,13 +4,15 @@ use warnings; use Data::Printer; + +use lib './lib'; use MetaCPAN::Util qw( es ); my $author = shift @ARGV; die "usage: $0 PAUSEID" if !$author; my $latest = es()->search( - index => 'v0', + index => 'cpan', type => 'release', fields => [ 'distribution', 'provides', 'version' ], size => 500, @@ -20,13 +22,13 @@ query => { match_all => {} }, filter => { and => [ - { term => { 'release.status' => 'latest' } }, - { term => { 'release.author' => [$author] }, }, + { term => { 'status' => 'latest' } }, + { term => { 'author' => $author }, }, ], }, }, }, - sort => [ { 'release.date' => 'desc' } ], + sort => [ { 'date' => 'desc' } ], }, ); diff --git a/scripts/release/6-latest-releases-with-git-repo-es.pl b/scripts/release/6-latest-releases-with-git-repo-es.pl new file mode 100755 index 0000000..42b7bea --- /dev/null +++ b/scripts/release/6-latest-releases-with-git-repo-es.pl @@ -0,0 +1,33 @@ +#!/usr/bin/env perl + +use strict; +use warnings; +use feature qw( say ); + +use Data::Printer; +use lib './lib'; +use MetaCPAN::Util qw( es ); + +my @must = ( + { term => { 'resources.repository.type' => 'git' }, }, + { term => { status => 'latest' } }, + { term => { authorized => 'true' } }, +); + +my $scroller = es()->scroll_helper( + body => { + query => { + bool => { must => \@must }, + }, + }, + _source => [ 'author', 'date', 'distribution', 'name', 'resources' ], + search_type => 'scan', + scroll => '5m', + index => 'cpan', + type => 'release', + size => 500, +); + +while ( my $result = $scroller->next ) { + my $release = $result->{_source}; +} diff --git a/scripts/endpoints/release/7-all-releases.pl b/scripts/release/7-all-releases-es.pl similarity index 66% rename from scripts/endpoints/release/7-all-releases.pl rename to scripts/release/7-all-releases-es.pl index e0e3529..80ecb56 100755 --- a/scripts/endpoints/release/7-all-releases.pl +++ b/scripts/release/7-all-releases-es.pl @@ -5,20 +5,17 @@ use feature qw( say ); use Data::Printer; + +use lib './lib'; use MetaCPAN::Util qw( es ); my $scroller = es()->scroll_helper( search_type => 'scan', scroll => '5m', - index => 'v0', + index => 'cpan', type => 'release', - size => 1000, - body => { - query => { - match_all => {} - }, - fields => ['download_url'], - } + size => 1_000, + body => { fields => ['download_url'] }, ); my @urls; diff --git a/scripts/release/7-all-releases.pl b/scripts/release/7-all-releases.pl new file mode 100755 index 0000000..c00dbfd --- /dev/null +++ b/scripts/release/7-all-releases.pl @@ -0,0 +1,12 @@ +use strict; +use warnings; +use feature qw( say ); + +use MetaCPAN::Client (); + +my $mc = MetaCPAN::Client->new; +my $release_results = $mc->all('releases'); + +while ( my $release = $release_results->next ) { + say $release->download_url; +} diff --git a/scripts/endpoints/release/all-releases-by-author.pl b/scripts/release/8-all-releases-by-author-es.pl similarity index 75% rename from scripts/endpoints/release/all-releases-by-author.pl rename to scripts/release/8-all-releases-by-author-es.pl index f720cf2..ede3d9a 100755 --- a/scripts/endpoints/release/all-releases-by-author.pl +++ b/scripts/release/8-all-releases-by-author-es.pl @@ -4,16 +4,16 @@ use warnings; use feature qw( say ); +use lib './lib'; use MetaCPAN::Util qw( es ); my $uploads = es()->search( - index => 'v0', + index => 'cpan', type => 'release', body => { query => { filtered => { - query => { match_all => {} }, - filter => { term => { 'release.author' => 'OALDERS' } }, + filter => { term => { 'author' => 'OALDERS' } }, }, }, fields => [ 'author', 'archive', 'date' ], diff --git a/scripts/release/9-all-releases-excluding-backpan.pl b/scripts/release/9-all-releases-excluding-backpan.pl new file mode 100755 index 0000000..4418ae6 --- /dev/null +++ b/scripts/release/9-all-releases-excluding-backpan.pl @@ -0,0 +1,12 @@ +use strict; +use warnings; +use feature qw( say ); + +use MetaCPAN::Client (); + +my $mc = MetaCPAN::Client->new; +my $release_results = $mc->release( { not => { status => 'backpan' } } ); + +while ( my $release = $release_results->next ) { + say $release->download_url; +} diff --git a/scripts/reverse_dependencies/by-dist-www-mech.pl b/scripts/reverse_dependencies/by-dist-www-mech.pl new file mode 100755 index 0000000..c1196d8 --- /dev/null +++ b/scripts/reverse_dependencies/by-dist-www-mech.pl @@ -0,0 +1,19 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +use Data::Printer; +use JSON::MaybeXS qw( decode_json ); +use WWW::Mechanize::GZip (); + +my $mech = WWW::Mechanize::GZip->new; + +$mech->get( + "https://fastapi.metacpan.org/v1/reverse_dependencies/dist/Carton"); + +my $results = decode_json( $mech->content ); + +my @dists + = map { $_->{distribution} } @{ $results->{data} }; +p @dists; diff --git a/scripts/reverse_dependencies/by-module-www-mech.pl b/scripts/reverse_dependencies/by-module-www-mech.pl new file mode 100755 index 0000000..c13dca1 --- /dev/null +++ b/scripts/reverse_dependencies/by-module-www-mech.pl @@ -0,0 +1,20 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +use Data::Printer; +use JSON::MaybeXS qw( decode_json ); +use WWW::Mechanize::GZip (); + +my $mech = WWW::Mechanize::GZip->new; + +$mech->get( + "https://fastapi.metacpan.org/v1/reverse_dependencies/module/WWW::Mechanize" +); + +my $results = decode_json( $mech->content ); + +my @dists + = map { $_->{distribution} } @{ $results->{data} }; +p @dists; diff --git a/scripts/endpoints/search/autocomplete/www-mech.pl b/scripts/search/autocomplete/www-mech.pl similarity index 80% rename from scripts/endpoints/search/autocomplete/www-mech.pl rename to scripts/search/autocomplete/www-mech.pl index 73ab2e5..ecf34e7 100755 --- a/scripts/endpoints/search/autocomplete/www-mech.pl +++ b/scripts/search/autocomplete/www-mech.pl @@ -5,14 +5,15 @@ use feature qw( say ); use Data::Printer; -use JSON qw( decode_json ); -use WWW::Mechanize::GZip; +use JSON::MaybeXS qw( decode_json ); +use WWW::Mechanize::GZip (); my $mech = WWW::Mechanize::GZip->new; my $search_term = shift @ARGV || 'HTML::Re'; -$mech->get("http://api.metacpan.org/v0/search/autocomplete?q=$search_term"); +$mech->get( + "https://fastapi.metacpan.org/v1/search/autocomplete?q=$search_term"); say $mech->content; my $results = decode_json( $mech->content );