Encyclopaedia Metallum: The Metal Archives

Message board

* FAQ    * Search   * Register   * Login 



Reply to topic
Author Message Previous topic | Next topic
Phate666
Mallcore Kid

Joined: Fri Sep 09, 2011 11:13 am
Posts: 12
PostPosted: Fri Aug 21, 2015 4:58 am 
 

Hi,

It would be really nice to have some kind of API (Rest or similar) to access the data of this website to simplify writing applications which use it.
Shouldn't be that complicated to implement. Another way would be to just release a (e.g. weekly) database dump, of course without any sensitive information like passwords etc.

I would love to get a response if this would be possible at all.

Top
 Profile  
Alhadis
Madder Max

Joined: Fri Sep 05, 2008 8:35 am
Posts: 4014
Location: Australia
PostPosted: Mon Aug 24, 2015 8:21 pm 
 

Similar questions have been brought up before. I believe it's on the cards but it's ultimately up to the webmasters to implement it.

I'm not sure where it sits on their priorities list, though.

Top
 Profile  
Phate666
Mallcore Kid

Joined: Fri Sep 09, 2011 11:13 am
Posts: 12
PostPosted: Fri Aug 28, 2015 3:26 am 
 

Judging by the lack of responses I can only guess they don't care. It's not like it would be that time-consuming to implement.
For some parts of the database there already exists a JSON API (e.g. artist lists by letters, review lists for artists). If I had access to the source code I could probably implement the missing stuff in less than a day.

Top
 Profile  
oneyoudontknow
Cum insantientibus furere necesse est.

Joined: Sun May 21, 2006 6:25 pm
Posts: 5078
Location: Germany
PostPosted: Thu Sep 03, 2015 10:16 am 
 

I think you are not the only one who would like to mess around with the stuff that is in the database.

I have done so several times already and did so last week again. I have now a set of data that displays the genres per country, dominant words in genres per country and dominant words on the whole. Sadly, obtaining this type of information was not easy. Interesting results though.

It would be nice to explore how the use of certain words have evolved over the years.
_________________

My website which contains reviews as well as interviews:
http://adsol.oneyoudontknow.com/index.php/wiki/A_dead_spot_of_light
Some analysis on the metal scene (data taken from the Metal Archives):
http://oneyoudontknow.com

Top
 Profile  
Alhadis
Madder Max

Joined: Fri Sep 05, 2008 8:35 am
Posts: 4014
Location: Australia
PostPosted: Sun Sep 06, 2015 2:12 am 
 

Phate666 wrote:
Judging by the lack of responses I can only guess they don't care. It's not like it would be that time-consuming to implement.
For some parts of the database there already exists a JSON API (e.g. artist lists by letters, review lists for artists). If I had access to the source code I could probably implement the missing stuff in less than a day.

Dude, it's probably less to do with giving a crap than it is maintaining a different set of endpoints to cater to a small minority of users. Yes, RESTful APIs have been requested in the past, but it's not like every second visitor to the site is hungrily requesting it.

Obviously implementing an API isn't rocket surgery, but it does add another layer of responsibility to maintaining the site's data model.

Top
 Profile  
Phate666
Mallcore Kid

Joined: Fri Sep 09, 2011 11:13 am
Posts: 12
PostPosted: Mon Sep 07, 2015 4:43 am 
 

Alhadis wrote:
Phate666 wrote:
Judging by the lack of responses I can only guess they don't care. It's not like it would be that time-consuming to implement.
For some parts of the database there already exists a JSON API (e.g. artist lists by letters, review lists for artists). If I had access to the source code I could probably implement the missing stuff in less than a day.

Dude, it's probably less to do with giving a crap than it is maintaining a different set of endpoints to cater to a small minority of users. Yes, RESTful APIs have been requested in the past, but it's not like every second visitor to the site is hungrily requesting it.

Obviously implementing an API isn't rocket surgery, but it does add another layer of responsibility to maintaining the site's data model.


As I mentioned before there already exists a JSON API for some parts of the data. Every single one of those sortable tables (e.g. http://www.metal-archives.com/lists/A) gets their data from there. There also already exists the code for the HTML output of the data, they simply have to add additional JSON or XML output.

Top
 Profile  
Alhadis
Madder Max

Joined: Fri Sep 05, 2008 8:35 am
Posts: 4014
Location: Australia
PostPosted: Mon Sep 07, 2015 10:19 pm 
 

Phate666 wrote:
As I mentioned before there already exists a JSON API for some parts of the data. Every single one of those sortable tables (e.g. http://www.metal-archives.com/lists/A) gets their data from there. There also already exists the code for the HTML output of the data, they simply have to add additional JSON or XML output.

First of all, you're aware of the CORS issues involved with requesting cross-domain payloads, right? :p

Secondly, those endpoints were written for internal use by the site; if they have to change at any point in future (which I'm goddamn sure they will), any code relying on it will break. They're not so much an "API" insofar as they are just a method of data exchange between the site's front and back-end. At no point do I imagine the webmasters intended for them to be processed by external sources. :p

Top
 Profile  
Phate666
Mallcore Kid

Joined: Fri Sep 09, 2011 11:13 am
Posts: 12
PostPosted: Wed Sep 09, 2015 5:38 am 
 

Alhadis wrote:
Phate666 wrote:
As I mentioned before there already exists a JSON API for some parts of the data. Every single one of those sortable tables (e.g. http://www.metal-archives.com/lists/A) gets their data from there. There also already exists the code for the HTML output of the data, they simply have to add additional JSON or XML output.

First of all, you're aware of the CORS issues involved with requesting cross-domain payloads, right? :p

Secondly, those endpoints were written for internal use by the site; if they have to change at any point in future (which I'm goddamn sure they will), any code relying on it will break. They're not so much an "API" insofar as they are just a method of data exchange between the site's front and back-end. At no point do I imagine the webmasters intended for them to be processed by external sources. :p


That is exactly the reason why I asked if an API is planned. I have already written a simple HTML parser (only several 100 LOC, took maybe 1 or 2 hours) to download all bands, releases and reviews. If they change something in the layout of the website this parser will most likely stop working.

Top
 Profile  
Alhadis
Madder Max

Joined: Fri Sep 05, 2008 8:35 am
Posts: 4014
Location: Australia
PostPosted: Wed Sep 09, 2015 6:55 am 
 

Well until then, your best bet is to stick with a headless browser or something, and play it safe with document.querySelector (or if you're a n00b and/or hate typing, jQuery :p).

Top
 Profile  
acidmartin
Mallcore Kid

Joined: Mon May 16, 2005 9:39 am
Posts: 2
PostPosted: Wed Nov 23, 2016 8:37 am 
 

Recently I published a 3rd party RESTFul API for Encyclopaedia Metallum. Currently it has 9 methods, but I am planning to add more soon. You can check it here: http://em.wemakesites.net/

Top
 Profile  
jasniec
Mallcore Kid

Joined: Sun Jul 08, 2018 9:55 am
Posts: 1
Location: Poland
PostPosted: Sat Nov 02, 2019 11:08 am 
 

Hi, I'm creating web scrapping library in C# / .NET.
I'm going to implement most features with bands / albums and songs.

The library is available on github: https://github.com/jasniec/MetalArchivesNET

Top
 Profile  
Display posts from previous:  Sort by  
Reply to topic


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

 
Jump to:  

Back to the Encyclopaedia Metallum


Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group