To remove directories, you have two options. You can remove all the objects inside the directory and then use rmdir to remove the directory itself:

$ mkdir mydir
$ touch mydir/file1
$ rm mydir/file1
$ rmdir mydir