Extracting data for browser bot

[Utharnl]

Hey everyone I have some problems extracting data for my browser bot (for Grepolis).
I wanted to extract information about the army training queue but this is saved in the html file like this:

<script type="text/html" id="orders_tmpl">
/*<![CDATA[*/
<h4>
	<% if(barracks) { %>
		In training	<% } else { %>
		Construction queue	<% } %>
	(<span id="current_building_order_queue_count">0</span>/7)
</h4>
<% for(var i in orders) { 
	var order = orders[i]; %>
		<div id="unit_order_<%=i%>" class="unit_order_task">
			<div style="position: relative;">		
				<div class="unit_order_task_time">
				<%=readableSeconds(order.to_be_completed_at-order.created_at)%>
				</div>
				<img src="<%=Game.img()%>/game/units/<%=order.unit_id%>_50x50.png" class="unit_order_task_unit"/>
				<div class="unit_order_task_value bold"><%=order.units_left%></div>
				<% if(i == orders.length-1) { %>
					<a href="<%=url(null, 'cancel')%>" class="unit_order_cancel cancel"></a>
				<% } %>
			</div>
		</div>
<% } %>

<% if(orders.length > 0) { %>
	<div id="unit_order_task_right"></div>
<% } else { %>
	<% if(barracks) { %>
		No units in training	<% } else { %>
		No orders in the construction queue	<% } %>
<% } %>
/*]]>*/
</script>

Which means that the html is only generated when the page is loaded in your browser.
Is there any way I can extract this data?

~Uthar

[SmackJew]

It is not "generated". The browser requests it doing a HTTP GET request and then receives it from the server.

Hypertext Transfer Protocol - Wikipedia, the free encyclopedia

[Utharnl]

Thanks for the response SmackJew, but I do know what http requests are .
The html source code I posted above is a part of the server response I received after a HTTP GET request.

Unfortunately they are using a javascript templating technique. (This link gives some information about it: )
The problem is that the browser generates this locally so that I can't see the information yet when I get the server response.
Now I know that there are here some more developers here, so I was wondering if someone has a solution for this?

~Uthar

[hallamasch]

Quick Answer:
2 Ways

1 - use webbrowser control
2 - Figure out what the javascript does and than do the same things in your code.

Good luck

[Utharnl]

Thanks hallamasch I have some ideas now to get it working with the webbrowser control, will give it a try when I have time.

But I would also like to know how to get it working with javascript, could always become useful for future projects.
The data that I need from the javascript part is saved in the array "orders", do you know a way to read that?

~Uthar

[hallamasch]

Again a Quick answer.

Find the javascript code that generates the orders array.

Since Javascript is client side, the server needs to send the data for the orders array.
Which you can intercept and than build the same functionality that the javascript is doing to that data.

[Utharnl]

Found the related javascript code after some time and they are requesting the data with a Ajax.get(...) request from the server.
But haven't been able to capture the corresponding server response yet. I used FireBug for this but will try it later with WireShark, maybe that gives a better result.

//edit
Got the correct server request
The javascript code sends an XMLHttpRequest to the server and by adding

X-Requested-With: XMLHttpRequest

to my get request it works just fine.
Thanks for pointing me in the right direction.

~Uthar

[hallamasch]

No Problem, your welcome =)