Nice linux command to ignore lots of files

I've been working with git from command line lately and in this particular project we have a folder to store cache files (we use other cache stores in production) which gets crowded really quickly and doing a git status was not pleasant at all, here is a screenshot from inside that folder (file/cache):

at the end of the list it would show the files I was interested in but still... A couple of minutes later I came up with this one liner to remove all of these files:

git status | egrep cache | tr -s ' ' | cut -d ' ' -f 2 | xargs git rm --cached $1;

Let me explain why it works, the first part (git status) runs the normal git status command but this one is cluttered with extra information, I just want the files that belong to the cache folder so a simple egrep cache filters the lines that do not contain the word "cache". Now, I wasn't sure if those were spaces or tab characters so to make it more universal I decided to remove duplicated spaces with tr -s ' ', it does not remove all spaces, just the duplicated ones so '    ' becomes ' '. The next part is the cut -d ' ' -f 2 which basically extracts the information from column 2, and finally pass that to git rm --cached 

Note that all of these commands are "glued" with the pipe character ( this one: | ) which takes the output from the previous command and passes it as a param to the next command.

Password Cracking with Hashcat

Disclaimer: Anything posted here is in the spirits of education, and education only. I am not responsible for what you do with the information here posted.

Ever since I attended Siren's talk on DDoS (she totally rocks btw!) I got -once again- interested in security and today I undusted an old proyect I worked on and tried to log in, no luck try after try until eventually I thought "well I can of course just reset it... or have some fun and crack it", after all, it uses md5 and its been a while since md5 was first cracked... shouldn't be too hard to find my old password right? well sure there are tools but the processing power available to me still makes it an arduous task... to make things worse the hashing format isn't plain md5, its a triple md5-d password, so the process is not straightforward.

Enter Hashcat, a very robust tool to crack passwords, their documentation wasn't dummy-proof and even after reading examples, their wiki and the help command I couldn't get anything working, fortunately "Xanadrel" in IRC helped me through and shortly after I got a better understanding of how it works. So, fair warning, I am not a cracker, Im not a super smart guy, I couldn't even get this to work on my own in the first try. What Im sharing here is what I learned today.

Password cracking is slow...

Im doing this in a virtual machine with 4 cores available and 3.3Ghz, my original password would take 16hrs assuming Im right and the last 2 characters are digits. Im sure putting more cores to work would highly improve the processing time but Im not going to do that.

Mask

The mask tells the format of the original password and is great if we know something about the password we are trying to crack like the length and the type of characters in the password. If we know that in a certain position of the password there is a number, we can tell Hashcat about it and the time it takes to find the password is reduced. Here is a nice table with the replacements:

What we know	Replace with
its a number	?d
its a upper case letter	?u
its an lower case letter	?l
its a symbol like <space>!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~	?s
any of the cases above	?a

Here are some examples of masks for specific passwords:

Password	Mask
14705	?d?d?d?d?d
azctqq	?l?l?l?l?l?l
1s5f7u	?d?l?d?l?d?l
! 4L	?s?s?d?u

Attack Types

Since we want to make this right, and we know something about the password, we can define a specific type of attack, this is the algorithm that Hashcat will use to find the password. Since we know something about the original password we can use the mask and because of this we need to specify a "brute force" attack, in reality this isn't a brute force attack but we need to define it as such (this was explained to me by Xanadrel). 

Depending on your operating system the way to pass params varies, since Im in linux this is how we pass the param for brute force:

-a 3

I believe in windows you need to use the long format.

Hash Types

There are many combinations to using md5, they call this "hash types", in this case since we know the password is a triple md5-d hash, we can find that in the references (run hashcat with the param --help and find the "References" section) with the number 3500 so we would add this param to the equation:

-m 3500

Increment

Hashcat assumes that if we set a mask of 7 characters we also want to check for passwords with 6 characters and less, if we know the exact length or even an approximate length it will be very useful to tell hc about it, we do this with the pm-min and pm-max params:

--pm-min=6 --pm-max=7

The Recipe

Hashcat reads hashes from a file, so make sure to put your hash in a file, it doesnt need anything special or fancy, just a one line with your hash and you're good to go, lets say you named this file "to_crack.txt", the final command would look like this:

shell> ./hashcat-cli64.bin -m 3500 -a 3 -show --pw-min=7 --pw-max=7 to_crack.txt ?a?a?a?a?a?d?d

I didn't need any salts but if you do, you add them to the end of your hash in your hash file so it would look like this:

ed1791de507c63335e735bd6ce7cd7bb:salt

The format is:

<hash>:<salt>

(One per line)

The output isn't all bells and whistles so you might miss where it says that it found your password, just look for the message "All hashes have been recovered" and above it you will find the hash and the password.

So there you have it, hope this helps you understand a bit how this magic tool works. If you have any improvements or comments in general feel free to post in the comments.

mysqldump: table doesn't exist

Today I needed to dump a database, nothing special, until mysql dump growled a "table <x> doesn't exist when using LOCK TABLES", ran a check

~$ mysqlcheck -udb_user -p database_name

but everything showed up fine... then I thought "well lets not lock it...", so this worked for me:

~$ mysqldump --skip-comments --add-drop-table --skip-lock-tables --user=my_user --password=my_password database_name >> db.sql

This database in particular was imported by copying files, I fixed the permissions and owner and it is working fine otherwise so I don't know what could be causing this problem which is why I dont count this as a "fix" but just a workaround, wiser folks may understand better what is happening and enlighten me.

MongoDB Noncopyable error

I was fiddling today with a C++ spaghetti and ran into this error:

/usr/include/boost/noncopyable.hpp|27|error: ‘boost::noncopyable_::noncopyable::noncopyable(const boost::noncopyable_::noncopyable&)’ is private

What it comes down to is that the mongo::DBClientConnection cannot be copied, we have to pass it by reference, like this:

void create_connection()
{
    mongo::DBClientConnection c;
    c.connect("localhost");
    do_something(c);
}

void do_something(mongo::DBClientConnection & c)
{
    // do something with the connection
}

Random Characters in NodeJS

Quick example on how to generate random characters in NodeJS

var crypto = require('crypto');
var randomChars = crypto.randomBytes(50).toString('base64');

That makes a synchronous call to randomBytes, so keep that in mind. If you want asynchronous then pass a second param to randomBytes:

var crypto = require('crypto');
crypto.randomBytes(70, function(err, buf){
 console.log(buf.toString('base64'));
});

more info can be found here

Recently (version 3.6) we moved all JS files to the end of <body>, this improves performance but there are some cases where you as a developer need to put JS files in <head>, in 3.7 we are introducing a way to do this, from a controller you can now do this:

<?php
$this->template()->setHeader('head', array(
    'somefile.js' => 'static_script',
    'otherfile.js' => 'style_script'
));

And it will be loaded in the head. Notice that the difference is the first param when calling ->setHeader.

Optimizing Phpfox - Tip #2

The feed seems to be the most resource hungry feature so far. If Timeline is enabled , when going to a profile the script needs to find which years have content so it can display the Timeline years block; this can span up to 20 queries to the database in some cases and while it makes proper use of indexes and conditions this is a load we can save in 2 ways:

1) Disable Timeline
2) Disable the Time block, this is perhaps the most feasible option, if you have timeline enabled it must be for a reason. To disable this specific block go to AdminCP -> CMS -> Block Manager, then click on profile.index and disable "Feed Timeline":

 you will keep the timeline look but the year selector wont be there

With this small change your site will be using a lot less resources and it will contribute to keeping it more stable and efficient.

Note: In case you are wondering, the queries that come from this block do in fact get cached, one cache file per user, but the cache file (specific to a user) is deleted after that user posts something that creates a feed.