I have been accused of censorship on my blog for deleting comments which are seen as criticism of my views on Telkom's technical flaws, and yes, I have been deleting comments. I have felt these comments attack are attacking to me as the author and that they are not discussing the issues at hand and I do not want anger, fighting or hate on this blog. This is a place to help, not a place to break down. If the authors feel they have a point, they are welcome to clearly and calmly rephrase it and post again. Alternatively, they are also welcome to post to Reddit where most of this has already been discussed and I have no control of what appears.
This is a follow up to my post on the man-in-the-middle attack that Telkom continues to use, as well as to the Telkom response in the awesome TechCentral article and new thoughts brought about by the Reddit post.
A real threat
Shortly after I posted my article I was contacted by someone (let us call them Person X ) who went snooping based on my post and found that Telkom did indeed have a major security flaw in the system. We agreed to hold on sharing the information publically until Telkom fixed it or enough time had elapsed to show they didn’t care. Telkom appear to have fixed it, as far as I can tell, so let us dig into this major attack vector.
Recapping the content that is served to you is made up of three pieces:
- HTML – bit of HTML is loaded, once again from the same server as above.
- Image – the graph showing you the usage is a static image served from a server which is (oddly) available everywhere: http://images.telkomsa.net/
Person X, realised that the images came from a folder ibn and that folder had directory listing enabled, which isn’t a good practise but, normally, not a major security concern. What the directory listing showed, besides the files, is the version number of the server which was (at the time) 2.0.52.
Apache 2.0.52 was released in November of 2004… 11 years ago and has NUMEROUS security flaws. There is no reason to run this version at all, it is insecure and points out a major security procedure flaw in Telkom. Using one of the flaws, it was possible to replace the specific image on the server with anything you wanted. For example, you could swop the image file with a flash file that would enable you to use one of the recent zero day attacks against flash and leveraging Telkom’s system to deliver said flash file to the user for you thus allowing you to own the target machine.
Remember this popup only shows to people who meet the following three requirements:
- Telkom ISP client
- At or near their limit
- Have not opted out
I would assume that the tech savvy of Telkom's users have opted out, and those near their limit are soft capped, meaning that security patches are slower to get to them. In short, it is the most vulnerable group who would be targeted. Scary right?! Let us hope Telkom fixed all the issues with that server and all the others.
Comments on Telkom’s choice wording
I am really happy Telkom has responded and is fixing things, that is all I wanted. That said, their choice of wording and delay on commenting until they fixed the issue is interesting. Let us break down their response and I am avoiding nit-picking since I could do a lot of that too.
In technical terms, we refer to it as an HTTP redirect
That is true. It is an HTTP redirect, but the manner it is used in is not the traditional sense of an HTTP redirect where the destination server tells the client to go somewhere else. Here a man in the middle (see what I did there ) is doing the telling, and thus I chose the term MITM attack as the description. It is not an exact description, but it is a description that describes the entire scenario and not just one choice aspect.
HTTP redirect is a common mechanism used in service provider networks for content caching and to optimise video streaming
does not alter the Web service content
is not a security risk
See above and then realise it was only true when it was said, not the day before.
will not ‘break’ a website
The web is a big place, it is impossible to know that. I also wonder why break is in quotes – is there more than one definition of break?
I have been asked recently about the ES6 support in VS and I haven’t had a good answer about how much of it is supported right now (i.e. VS 2015 RC) and what is supported. The general feeling from the askers is that VS is far behind in this space, so the only reasonable thing to do is for me to test this and let you know.
Using Luke Hoban’s awesome page on ES6 features gave me a great point to kick off from around the different features. I then tested each feature in VS 2015 RC and also in VS Code (our new lightweight, cross-platform, free IDE) and in the end Visual Studio has 70% implemented and Code has 94% implemented. What does implemented mean though? It means no errors & and the IDE does the right IntelliSense stuff for you. There is on-going work with both IDE’s so this will improve but as a baseline for discussion it is useful.
The way I worked it out is to assign a 1 (works) or zero (doesn’t work) to each feature. In some cases I assigned a middle value because it kinda worked and the footnotes will explain that. The only missing one on here is the reflect API. I don’t have an example I can work with around that yet, so I did not include it. If you find any errors here, please let me know and I will gladly update!
Enhanced Object Literals
Default + Rest + Spread
Let + Const
Iterators + For..Of
Map + Set + WeakMap + WeakSet
Math + Number + String + Array + Object APIs
Binary and Octal Literals
- No issues in the IDE are raised but the IntelliSense is lacking.
- No issues in the IDE are raised but the IntelliSense is lacking.
- There is an odd issue where an extra warning is raised in the wrong place. It doesn’t break anything but isn’t ideal.
- No IntelliSense on the loaders.
- Again lack of IntelliSense is the issue.
- Weird warning appearing, which looks like a bug.
In February this year I was contacted by the team at PACKT Publishing about being a technical reviewer for a book which was underway, in exchange I would get a free copy of the book and be listed in the book as one of the reviewers. I have toyed with the idea of writing a book for ages so this felt like a good idea to see what happens behind the scenes without committing to actually writing a book. I said yup to them, and today that book got published! You can get it at: http://bit.ly/json_cookbook (I am really excited about this)
The book* itself is a very interesting mix of content, from the very basics of JSON to introduction to MongoDB and storing data in it. It very much hits the nail on the head with the description “Quick answers to common problems”. What I think is really awesome about it, is that it really tries to cover a lot of languages & tools. So there is .NET, Java, Node.js, Android, ObjectiveC and more. The examples are really great as all work on a variety of OS’s too, so you can quickly try stuff out. I don’t think it is a book you It isn’t a book you would read end to end but rather give you guidance on where to start with problems related to JSON.
The experience itself of reviewing was interesting, each chapter took a few hours of reading and trying out the code and responding with details of issues found. The people I worked with at PACKT made it really pleasant. I never heard from the author or other reviewers, which in hindsight is odd but I think that maybe helped keep my responses focused and not have a bias in them.
If you do pick it up, let me know what you thought of it!
*Note: I’ve only see the content I reviewed, I haven’t seen the final book which maybe different.
Vorlon.js is an amazing tool for web developers as it brings the browser developer tools, aka F12, out of the limitations of a single browser and to the cloud. While you can run it on your machine, I believe to really get the full power of it, you need to put it on a dev server somewhere. The broader the reach, the more use. So with that mindset here is how to run it on an Ubuntu server and for this example I am use Azure to host that VM. I am using the Basic A2 (2 Cores, 3.5 GB) machine which is more than enough for Vorlon and means I can run the server pretty cheaply.
Step 1: Get NPM
Vorlon uses NPM, Node Package Manager, for distribution so you need that to get started. Before you begin, make sure you are up to date:
sudo apt-get update
Once that is done you can run the following to install NPM:
sudo apt-get install npm
If you didn’t follow my advice on doing the update first, or you ran into issues, you may need to run the command again with extra parameters:
Step 2: Get Node.js
Vorlon is built with Node.js, so you need to get that too. This took me surprisingly long to figure out, but thanks to this page I came right. The commands to run are:
curl --silent --location https://deb.nodesource.com/setup_0.12 | sudo bash - sudo apt-get install --yes nodejs
Step 3: Get Vorlon
Now that the machine is setup correctly, you can get a copy of Vorlon using NPM with the next command:
sudo npm i -g vorlon
Step 4: Run Vorlon
Running Vorlon is easy, just type: vorlon
For Windows users, make sure it is all lower case. It takes about 30secs and you will see the command output pop up
Once that shows up you can access it!
/usr/bin/env: node: No such file or directory
Every time I ran Vorlon originally I got the following error message: /usr/bin/env: node: No such file or directory
This is because I didn’t have Node.js installed. I had npm, which I assumed brought Node.js with it but it doesn’t and so Vorlon won’t run. In hindsight it is obvious, and even the error message tells me that but at the time – well it was an hour to figure out :/
Azure – don’t forget those endpoints
If you are on Azure, you will also need to setup the endpoints to allow access to the VM. To do this, go to the portal and open you VM. Click All Settings, then Endpoints, then Add. From there you need to add an endpoint that maps the public to the private. I keep both the same (1337) but you could make the public something different.
You want to code in TypeScript. You want to code with Sublime Text. Today, that is really really easy to do, thanks to Package Control and Microsoft, who is now providing a first class plug-in for Sublime Text to enable & light up TypeScript.
Step 1: Install Package Control
The installation instructions for it can be found here, but in summary you press Ctrl+` and then paste the following code (for Sublime Text 2, for 3 see their website) into the text box and press enter
import urllib2,os,hashlib; h = 'eb2297e1a458f27d836c04bb0cbaf282' + 'd0e7a3098092775ccb37ca9d6b2e4b7d'; pf = 'Package Control.sublime-package'; ipp = sublime.installed_packages_path(); os.makedirs( ipp ) if not os.path.exists(ipp) else None; urllib2.install_opener( urllib2.build_opener( urllib2.ProxyHandler()) ); by = urllib2.urlopen( 'http://packagecontrol.io/' + pf.replace(' ', '%20')).read(); dh = hashlib.sha256(by).hexdigest(); open( os.path.join( ipp, pf), 'wb' ).write(by) if dh == h else None; print('Error validating download (got %s instead of %s), please try manual install' % (dh, h) if dh != h else 'Please restart Sublime Text to finish installation')
It takes a few seconds where nothing happens and then, without much fanfare, the message window will say “Please restart Sublime Text to finish installation” – so do that.
Step 2: Add the TypeScript package
Press Ctrl+Shift+P to bring up the command palette and type in Package Control: Install Package and press enter.
You can use the arrow keys to navigate the list quicker and press enter on the right item rather than typing everything.
In the next command palette window type TypeScript and hit enter.
Once that is done, it is done and you will get glorious TypeScript in Sublime!
Part 2 is here, click that now (it will open in another window) so you can continue the fun when you are finished here
The following image is a screen shot I took of a webpage in Chrome. If you view that same website, you will get a different experience because Telkom is using the same techniques as a man-in-the-middle attack to edit the code which the website uses, to serve Telkom’s own content.
The important bit which has been added, is the piece in the lower right hand corner of the page, which Telkom has added to the website without the permission of the website owner or the customer of the Telkom ISP account.
Admittedly this is a relatively benign addition, in fact it maybe seen as useful, and I can see it being sold that way to non-technical managers and executives. Do not be fooled though, even this simple addition can cause major issues for you.
Breaking of websites
Exposing of security risks
The second issue is that they are exposing users to the potential of massive security risks. They may argue that their code is secure now and that maybe true, however that is a point in time view.
A second issue around the security, is that they need to host their content & systems which do this attack somewhere. If that server/s is compromised – either from an external attack or from a disgruntled employee they could use that for XSS attacks or key sniffing attack! As this “feature” becomes more widely known, you can be assured that those servers will become the target of more attacks because of their value of leveraging them to launching attacks on users will be very big. The reality is that it is just a matter of time before those servers are compromised.
Taking away your freedom to privacy
While it will always be possible for ISPs to see what the traffic you are sending & receiving is, it shouldn’t be easily accessible to the staff and people at the ISP. Having a server that can do this manipulation means that they are enabling a very easy point for someone to capture traffic and see what you are doing on the internet in such a way that it will be hidden from you.
While I am sure they will tell you they take security very seriously and that they do not allow that type of access to employees what is stopping an executive at a later stage from using this to prevent adverts from MTN showing up or causing web pages that support EFF or the DA to not load at all? Nothing and they have that power to do that, without oversight and without your permission. Do you trust Telkom enough to not abuse that?
Telkom doing this is violating a founding principal of the Internet: Net neutrality
Finally, and this is last because it is the least of the issues, they are making you download more 84.8Kb of extra code (see here for the data) and ~120Kb of extra images plus the manipulation of the web page slows down rendering. In short, they are making the web slower for you and helping use more of your bandwidth. They are also adding in extra servers and network hops which add even more performance concerns.
How are they able to do this? From my testing they only do this to HTTP traffic, not HTTPS traffic, as that traffic is encrypted. While the encryption should help, it is not impossible to silently break and inject content as shown with the recent Lenono Superfish disaster which is basically the same thing as Telkom is doing except on an ISP level. It isn't easy or likely reasonable for them to do that, and I am not implying they are, just that they could and maybe motivated to add this to secure websites.
To explain this it might help to know what a normal process of getting a file looks like:
- You web browser requests a file that is needed for the website to load the web page
- It hits the Telkom proxy, which is meant to help speed up the Internet by providing content to you from a closer location. They may not have a proxy, many ISPs do this so that they can provide a pure & secure experience.
- If the proxy doesn’t have the content, it should ask the web server for it.
- The web server provides the content to the proxy.
- The proxy returns it to the browser unaltered.
The way this attack works is as follows:
- You web browser requests a file that is needed for the website to load
- And the website provides it.
- The resulting modified code is sent back to the proxy
- The proxy returns the request to the browser.
Disclaimer: This is a rough view of it and without inside knowledge I cannot guarantee it is exactly like that. It maybe that there are changes but the principal will be the same.
Note: these URLs do not work outside the Telkom ADSL network.
Are you sure your OS / Browser isn’t attacked? Are you sure this is Telkom?
In the image above it is Chrome on Windows 10. I have been able to reproduce this on my Windows Phone and also using Microsoft’s new Edge browser. This is on the network layer
And here is it again on the jQuery website using Microsoft Edge:
How are they showing content without changing the HTML?
How can you be sure that this isn’t an agreement with the web site owner?
Yup. Here is the modified Modernizer from the PowerBase website: http://pastebin.com/1p4m5LnM
Here is the modified jQuery from StackOverflow: http://pastebin.com/XXY6b3n7
I am on Telkom ADSL and I am not seeing this, you are wrong!
- You need to have Telkom as your ADSL & ISP. If I switch my ISP to FNB, for example, this doesn’t happen.
- Second it doesn’t happen on HTTPS, or at least I’ve never seen it. StackOverflow is a good example where it would always happen on HTTP, but when I switched to HTTPS (they support both) it stopped.
- It isn’t constant. I first picked this up a few days ago and then it stopped until the weekend.
What can I do to protect myself?
You can’t block those servers as that will break the websites where Telkom is injecting content into. Telkom has a web page which allows you to opt-out. I haven't been able to confirm it works yet but I will be keeping an eye out for it. Another possible tool to help protect you, you could use the HTTPS Everywhere extension to increase the amount of websites you access with HTTPS, thus lowering the amount of this type of nonsense.
Ultimately, you need to trust your ISP. They can see your data and manipulate it. You need to trust they won't. If they show they are unwilling to live up to the principals of the Internet or violate your trust, you only option is to change ISPs.
What if Telkom lets me opt-out, that is good enough right?
Opting out means that they will still have this system. The system will just check the database to see if you get the original files or the modified files. You still are slowed down for the check (versus performance of compared no check at all because this system is taken out). In the best case (i.e. just a DB lookup and then a redirect), once opted out that performance difference will be so marginal it doesn't matter. In the worst case, they could still direct you through the same (potentially vulnerable servers) and you could get a lot more overhead due to that, bad code etc....
The only approach which introduces no further security issues, no violation of net neutrality and no performance impact is for Telkom not to do this at all.
As I am totally happy to admit when I am wrong, any updates will be listed here
12h05 10 July 2015 - I wrote many ISPs do not use proxies by mistake. I meant many do, but only for performance reasons. Not for changing of content and added more info & clarification on the HTTPS breaking.
15h00 29 June 2015 - it was pointed out to me it isn't just the JS you download, it is the MASSIVE image too. So rather than 80ish Kb it is almost 200Kb in total! I've updated the article to reflect that.
15h30 29 June 2015 - Added points reflecting that this is a violation of net neutrality and the HTTPS everywhere extension which might help some people.
18h25 20 June 2015 - Added information about opt-out option. Added information around performance and how it may or maybe impacted by opt-in out. All of that had some rewording to the "What can I do to protect myself?" section.
I try not to rant on my blog and mostly I succeed. Today though I wanted to complain to Telkom about them inserting content into web pages on Twitter so I head to their website to find their Twitter account and I don’t find one, I find THREE different Twitter accounts and a situation so bad I cannot keep shut!
It isn’t ideal to have three and I can understand different components need it, in fact they look like a business account (1), their cellphone brand (2) and a general one (3). The part that baffles is what happens when I checked them.
The business account
This account has been taken over by spammers :facepalm:
The mobile account
That account is suspended :facepalm:
In fact only the general account works! Shit happens, people mess up and accounts get suspended or lost – it isn’t ideal or professional and you can mitigate the damage by removing it from your website. The fact Telkom can’t/won’t speaks volumes to their inability to care. There is no acceptable reason for those bad links to be up there and who ever runs their website should hang up their HTML spurs and find a new career.
Let them know
Yesterday in Johannesburg we had an amazing //Build Tour event! We had top international speakers like Pete Brown join us and wow the audience with Windows 10, IoT, Edge and more! Unfortunately for the international speakers, they had flight issues and we had a hour and half gap before we could kick off, so a bunch of the local Microsoft people got together to provide a bit of local flavour in a pre-keynote event. I was very excited to share about Visual Studio 2015 and it got cut at the last minute as the presenters arrived! Part of my prep was to create a video of my talk and you can watch this content now:
You can get the demo code on GitHub.
With an agile team you should meet daily for a quick meeting. Normally that takes the form of each person answering a three question status update (What did I achieve, what will I achieve, what is blocking me). For a team working together it means these steps:
- Team arrives at the shared team board
- Team goes round and gives status update
- Team goes back to work
Just because your team is distributed doesn’t mean this shouldn’t happen – it just means that you’ll find a few extra steps:
- Team members who work together go to booked meeting room (you need a meeting room so those remote can hear over the normal work environment noise)
- Be surprised to find some one in the room
- Explain that you booked the room
- Wait for them to leave the room
- Connect the projector
- Realise you left your microphone in your laptop bag, return to desk get it, return to meeting room
- Plug it in
- Open the digital scrum board
- Start meeting software and join meeting
- See one remote team member is already there – apologise for being late
- wait for response
- wait for response
- realise they can’t hear you.
- Both the remote team member and yourself try to figure out what is not working
- Figure out that there is a mute hardware button that today (and today only) is turned itself on
- Try again – success, they can hear you
- Wait for the product owner to join.
- Give up waiting and kick off meeting (everyone is seated now cause no ones legs last that long)
- Go around the team and give status update
- Product owner joins during second last team members status update
- Product owner starts sharing a bunch of bugs and task updates that someone else is now capturing live
- Finish status updates
- Team decides to use this time that we are all here to discuss technical info around the new stuff from the product owner
- Product owner is late for meeting and has to disconnect
- Someone walks into the meeting room and asks if this is the meeting for organising the year end party
- No one answers
- Scrum master answers
- Team discusses technical issues. Half are marked as needing more info and will be discussed with product owner in tomorrows meeting
- Call ends
- Team goes to get coffee (they need it) and says how glad they are using agile cause they save so much time with less meetings
- two hours later
- other team member joins call alone because the time zone he is in changed to daylight savings today