Page 1 of 2

Replays (casted) database and "epic" matches

Posted: Sun 17 Feb, 2013 2:30 am
by Lulgrim
I have set up a replays database that lists Elite matches uploaded to YouTube with basic metadata and filtering options. Filtering by map, faction, hero etc. can be added if useful. The list is not complete yet yes it is, all Elite replays from the (major) channels are in. New casts are added within a few days as they go up.

One of the filtering options is "epic" which is a hall-of-fame type of thing, awarded to a match which is totally rad. I flagged one game so far. Suggestions for "epic" status can be posted in this thread.

http://www.dawnofwar.info/index.php?page=replays

Re: Replays (casted) database and "epic" matches

Posted: Thu 23 May, 2013 4:44 pm
by FiSH

Re: Replays (casted) database and "epic" matches

Posted: Thu 23 May, 2013 4:45 pm
by Lulgrim
Reasonable suggestion, epic'd.

Re: Replays (casted) database and "epic" matches

Posted: Mon 03 Jun, 2013 5:51 pm
by Bushido
http://www.youtube.com/watch?v=YbsOlT5ccPY

if this ain't epic, i don't know what is

Re: Replays (casted) database and "epic" matches

Posted: Sun 15 Sep, 2013 8:55 am
by Lulgrim
I updated the replay database this weekend after letting it fall behind for a few weeks and now we have reached 500 Elite replays over the three main YouTube channels (Dawn of War II Replays, Maestro Casts and FitzRupee Casts).

At the moment we are logging match date, title, map, duration, mod version and event (if it was part of a tournament or such, although this is not displayed yet) - but if someone has a real OCD with replays, we could save races/heroes as well (or even player names) in the database. However, all this is unavailable to any sort of automagic process so it would have to be entered manually. Hence the OCD... but if someone finds that interesting, contact me.

If there are other channels posting quality casts (HD video, good commentary) we can consider them for the database. YouTube would be easiest.

Re: Replays (casted) database and "epic" matches

Posted: Sun 15 Sep, 2013 9:49 am
by Lulgrim
I added functionality for filtering replays by map (use the 1v1, 2v2 and 3v3 buttons to filter 2p, 4p and 6p versions of maps like Calderis Refinery - this way the map list stays a bit shorter).

Re: Replays (casted) database and "epic" matches

Posted: Mon 16 Sep, 2013 7:40 am
by Lulgrim
I added an option to search/filter videos by event (e.g. tournament) although the filter selection area looks a bit stupid now. If there are games belonging to a tournament that were not flagged properly, let me know.

Re: Replays (casted) database and "epic" matches

Posted: Thu 03 Oct, 2013 3:28 pm
by Uncle Milty
Hi,

i don't know, if you have already seen it, but Eerie and me have set up a youtube channel for the monthly rumble tournaments, where i upload the games i have featured on my stream. Also Eerie wants to add some fancy stuff (later?).
Since you have a filter section for those tournaments, you might as well add those vids to your database.
Also you might want to add my stream to the streams list.

Re: Replays (casted) database and "epic" matches

Posted: Sun 03 Nov, 2013 7:09 pm
by Blessed Spartan
http://www.youtube.com/channel/UCh8alfs0qCzKtkF-ihG4pog

my youtube channel, hit it for some humorous casts ;)

Re: Replays (casted) database and "epic" matches

Posted: Fri 06 Dec, 2013 8:13 pm
by Arbit
Awesome stuff; I have a lot of casts to catch up on so I will make good use of this.

Are the replays stored somewhere we can download them, or are all these directly submitted to the caster?

Re: Replays (casted) database and "epic" matches

Posted: Fri 17 Jan, 2014 1:35 pm
by Lulgrim
Some go to casters, I guess some can be found on GR for example. There's no central archive for the actual replay files, although I still have all ranked 1v1 replays stored.

As for the replay database, I updated it with the Montly Rumble channel and Crazyman64335 and we now have more than 800 replays there.

Re: Replays (casted) database and "epic" matches

Posted: Sat 10 May, 2014 10:22 pm
by Lulgrim
Fixed some map search conditions bugging, fixed line breaks in the description text, and made better pagination links. Updated index, there's now 1087 videos in it.

Re: Replays (casted) database and "epic" matches

Posted: Sun 11 May, 2014 2:03 pm
by Tex
schwweeeeeeeeeettt. Dats a lot of vids man! Downtime viewing all set :D

Re: Replays (casted) database and "epic" matches

Posted: Tue 01 Jul, 2014 12:06 pm
by Chronoslayer
Always thought this was one of the most epic casted replays:

https://www.youtube.com/watch?v=hHkqM1NhNok

Re: Replays (casted) database and "epic" matches

Posted: Sun 17 Aug, 2014 3:47 pm
by Adeptus Noobus
Is there a way to filter the replays by factions? Sometimes I just want to find Space Marine replays or Eldar replays. Would be a feature/filter.

Re: Replays (casted) database and "epic" matches

Posted: Sat 06 Sep, 2014 3:55 pm
by Lulgrim
It's easy to code BUT it would require someone to enter more metadata in the replays database. If there's a volunteer for that (go through 1000+ reps and input hero for each player) then np I'll code it for you.

Re: Replays (casted) database and "epic" matches

Posted: Mon 08 Sep, 2014 1:13 pm
by Adeptus Noobus
I get what you mean. Would it create a conflict if only future replays would contain these new metadata. If yes, could not all old videos be simply linked to all factions, saving the time to put them all into the right factions.
Another possibility would be me going through these videos if you give me a list I can work with.

Re: Replays (casted) database and "epic" matches

Posted: Mon 08 Sep, 2014 5:23 pm
by Swift
That is essentially what Lulgrim said, if someone is willing to categorise the old replays by faction he can code the tool for it so someone can reorganise them all.

Re: Replays (casted) database and "epic" matches

Posted: Mon 13 Oct, 2014 8:43 pm
by Barrogh
No replay files, sadly :(

Re: Replays (casted) database and "epic" matches

Posted: Sun 07 Dec, 2014 5:08 pm
by Adeptus Noobus
Ok, I have found the time to look into this and Unix is actually (as usual) quite powerful for this problem.

I do not know the folder structure of the database in which the replay files are saved but here it goes:

Code: Select all

strings 6p_garvusicegate.2014-11-21.02-10-47.rec | grep -A1 DATAINFO | sed s/DATAINFO//g | sed "s/[^a-z]//g" | grep -v data | sort | uniq


This will print out a list of all the factions that were involved in the match. Each faction also only occurs once to avoid doubles etc. This would be troublesome if the replay were for example from a faction war.
The output looks as follows (in the case of my replay file):

Code: Select all

chaosmarines
eldar
imperialguard
spacemarines

Notice how the number of factions does not add up to 6, as multiple occurrences have been filtered out.
If however you wanted the full list in order of the players appearance, use this code instead:

Code: Select all

strings 6p_garvusicegate.2014-11-21.02-10-47.rec | grep -A1 DATAINFO | sed s/DATAINFO//g | sed "s/[^a-z]//g" | egrep -v "data|^$"

chaosmarines
spacemarines
spacemarines
eldar
imperialguard
chaosmarines


From here on, I believe it is very easy to copy each replay file into a faction-specific folder or something alike, to seperate the replays by faction.

I hope this helps.


EDIT:
The color schemes could be extracted in the same way if wanted. The players names however are more difficult to extract as Relic has chosen to code them in a rather funny way: not as a string but as sequences of letters divided by decimal points making it more difficult to extract them from the data mumbo jumbo.

Re: Replays (casted) database and "epic" matches

Posted: Sun 07 Dec, 2014 7:27 pm
by Adeptus Noobus
OK, after having conferred with the dark gods of coding (more bytes to the byte throne) I may have come up with a solution:

Code: Select all

hexdump -v -e '/2500 "%010_ad |"' -e '2500/1 "%_p" "|\n"' 6p_garvusicegate.2014-11-21.02-10-47.rec | sed $'s/DATAINFO/\\\nDATAINFO/g' | sed -n 's/.*DATAINFO\(.*data\).*/\1/p' | perl -pe 's/(\.\.\.\.)+/\|/g; s/^\|//g;s/\.//g' | perl -lne ' @A = split(/\|/,$_) ; $A[1] =~ s/[^a-z]//g; print "$A[0]|$A[1]" if $A[1]'


Twitchtv/Corteaz"!|chaosmarines
sk4zi|spacemarines
Adeptus Noobus|spacemarines
Ar-Aamon|eldar
TehMuffinman571|imperialguard
Lupercal|chaosmarines


The code dumps the header of the replay file -> Extracts the neccessary lines containing the information -> formats the output and voilá we have the race AND the player name.

This code has been tested on a MacBook under Mac OSX. The syntax may be slightly different for other unix systems.
But there you have it, a way to automatically extract the player names and the corresponding races. Sadly the heroes used are not coded in clear text, so that I can not (yet) account for them in the code.

Any feedback would be greatly appreciated.

EDIT
The last two characters in Corteaz' name are the result of him using special characters in his username. Unfortunately I can not yet convert these. I would have to account for every special character there is, which quite frankly, I dont want to do. :)

Re: Replays (casted) database and "epic" matches

Posted: Sun 07 Dec, 2014 8:57 pm
by Atlas
Just a really general question but is there some way we can implement an upload feature? I imagine it looking like a simple "insert file ->" upload and then it'll ask you to fill in details manually for you such as:

1v1/2v2/3v3
Factions used
Map
Description
etc etc

Yes, I'm essentially asking for a gamereplays.net specific to the site for Elite and people can just up/down load stuff from here.

Re: Replays (casted) database and "epic" matches

Posted: Wed 18 Mar, 2015 5:04 am
by Adeptus Noobus
Is this still of any interest to you Fulgrim?

Re: Replays (casted) database and "epic" matches

Posted: Wed 18 Mar, 2015 6:54 pm
by Myrdal
Barrogh wrote:No replay files, sadly :(

Idd, this would require the actual replay files from each cast :( Unless the casters have been super organized and stored the reply file or relevant data with the corresponding casted video, this is gonna require some major manual work.

Gotta love that unix-fu though.

edit: looks like at least Indrid writes down races/heroes in the video description so we could easily add this data to the db.

Re: Replays (casted) database and "epic" matches

Posted: Wed 18 Mar, 2015 11:58 pm
by Adeptus Noobus
I always thought that all the replay files were actually uploaded here. Now I understand what Fulgrim ment when he said it had to be done manually. Well if you consider adding a upload feature (like Atlas asked) to this site where people can simply upload their replays, you could use this script to extract all the data automatically (in addition to the info that you already extract i.d. chat transcript etc.) and wipe the files afterwards. This would make keeping the ranked list up-to-date pretty easy. Just an idea.

Re: Replays (casted) database and "epic" matches

Posted: Wed 25 Mar, 2015 8:48 pm
by Myrdal
Adeptus Noobus wrote:Well if you consider adding a upload feature (like Atlas asked) to this site where people can simply upload their replays, you could use this script to extract all the data automatically (in addition to the info that you already extract i.d. chat transcript etc.) and wipe the files afterwards. This would make keeping the ranked list up-to-date pretty easy. Just an idea.

It's a lot better than having 1 guy fill in the data but I doubt ppl will bother with this. The only acceptable solution is a automatic one I think.
Anyway I've made the Replay db able to self-update and I also made it parse players, heroes and map played from youtube but it will only work if casters fill this out (in rather consistent format). Torpid's channel was also added but please add the map in description :)
Basically all this effort in order to easily find casts with certain MU's, see http://dawnofwar.info/index.php?page=replays. Note older replays have not been fully indexed.

Re: Replays (casted) database and "epic" matches

Posted: Wed 25 Mar, 2015 9:52 pm
by Atlas
Pretty amazing bro, ty!

Re: Replays (casted) database and "epic" matches

Posted: Thu 26 Mar, 2015 11:18 am
by Adeptus Noobus
This is awesome. Maybe we should make all the casters aware of this format of yours so that they abide by it.

Re: Replays (casted) database and "epic" matches

Posted: Thu 26 Mar, 2015 10:16 pm
by Myrdal
No special format needed but it be nice if they kept it consistent, that way I don't have to update the reg-ex filters. I've indexed most of the replays now, but many will be missing data because there war errors parsing it or it just wasn't entered in the title/description.

Re: Replays (casted) database and "epic" matches

Posted: Sun 18 Oct, 2015 5:37 pm
by Adeptus Noobus
We have had quite a few tournaments since the last update to the ratings. Is there any way that we could access the ratings part of the site to update them? (Yes, I will do it, if you show me where I can do it)
The ideas we were throwing around to actually make changes to the replay files to include all the neccessary data did not bear any fruits so far I guess, because of the lack of programmers who actually have time to do so?